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FOREWORD 


*  In  1977,  the  Army  conducted  an  investigation  of  the  feasibility  of  using 
reduced-physical-fidelity  training  devices  for  maintenance  training  rather  than 
the  actual  equipment  then  in  service  (Durall,  Spears,  &  Prophet,  1978) .  The 
results  of  that  effort  indicated  that  reduced-physical-fidelity  trainers  were 
appropriate  for  training  a  number  of  maintenance  tasks.  The  positive  recom¬ 
mendation  for  reduced-physical-fidelityj  trainers  led  the  Army  to  embark  on  de¬ 
velopment  of  the  Army  Maintenance  Training  and  Evaluation  Simulation  System 
(AMTESS) ,  an  effort  designed  to  provide  the  Army  with  more  cost-  and  training- 
effective  mainteneuice  simulators. 


By  1981,  the  Army  had  received  two  prototype  maintenance  simulators.  In 
a  previous  report,  Criswell,  Unger,  Swezey,  and  Hays  (1983)  provided  an  his¬ 
torical  review  of  the  AMTESS  effort.  The  purpose  of  this  report  is  to  provide 
a  case  study  of  the  front-end  analytic  activities  which  led  to  the  construction 
of  prototype  AMTESS  simulators.  Based  on  this  case  study,  the  report  suggests 
improvements  in  the  front-end  analysis  procedures  which  can  be  used  to  guide 
future  AMTESS- like  efforts.  — —  - - - 
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A  CRITICAL  REVIEW  OF  FRONT-END  ANALYSIS  PROCEDURES  EMPLOYED  IN  THE 
ARMY  MAINTENANCE  TRAINING  AND  EVALUATION  SIMULATION  SYSTEM  (AMTESS) 
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EXECUTIVE  SUMMARY 


Requirement: 

The  Army  Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS) 
represents  an  initiative  to  develop  generic  maintenance  training  simulators 
which  are  cost  effective  and  training  effective.  The  purpose  of  this  report 
is  (1)  to  review  and  evaluate  the  front-end  analysis  (FEA)  procedures  performed 
in  Phase  I  of  the  AMTESS  project,  and  (2)  to  provide  a  series  of  general  guide¬ 
lines  for  conducting  front-end  analyses  for  future  maintenance  training  device 
development. 


Procedure: 

The  procedure  used  in  this  report  involved  a  thorough  review  of  existing 
AMTESS  documentation  as  well  as  pertinent  training  literature.  In  particular, 
three  types  of  documentation  were  reviewed  including  government  documentation 
(e.g.,  military  training  device  doctrine  and  related  studies),  AMTESS  require¬ 
ments  documents  and  reports,  and  pertinent  device  evaluation  reports.  The 
major  focus  on  this  review  was  upon  the  FEA  activities  performed  by  four  AMTESS 
contractors  (Grumman  Aerospace  Corporation,  Honeywell  Systems  and  Research  Cen¬ 
ter,  Hughes  Aircraft  Company,  and  a  consortium  composed  of  Seville  Research 
Corporation  and  Burtek,  Inc.)  and  by  the  Army.  This  review  of  FEA  activities 
was  guided  by  the  following  four  criteria:  (1)  clarity  of  requirements, 

(2)  completeness  of  performance,  (3)  effectiveness  of  coordination,  and 
(4)  preparation  for  device  construction  (AMTESS  Phase  II)  and  evaluation. 


Findings: 

With  respect  to  clarity  of  requirements,  it  was  found  that  all  four  con¬ 
tractors  developed  very  similar  device  concepts,  suggesting  that  the  initial 
requirements  were  clear.  The  major  area  in  which  the  AMTESS  1  Request  for 
Proposal  (RFP)  lacked  clear  guidance  was  task  selection  which,  in  turn,  had  a 
serious  impact  on  device  evaluation.  Also,  the  RFP  specification  that  AMTESS 
would  be  an  "adjunct"  to  and  "supplement"  conventional  maintenance  training 
appeared  to  cause  confusion  regarding  task  selection. 

In  terms  of  completeness  of  performance,  the  results  showed  that  two  con¬ 
tractors,  Grumman  Aerospace  Corporation,  and  Honeywell  Systems  and  Research 
Center,  produced  all  that  was  requested  in  the  AMTESS  Phase  I  effort.  Seville 
Research  Corporation  also  was  responsive,  with  the  exception  of  providing  a 
device  evaluation  plan  (although  Seville  previously  had  provided  such  a  plan 
in  its  preliminary  feasibility  study  prior  to  AMTESS  Phase  I).  Hughes’  per¬ 
formance  was  marked  by  a  task  analysis  which  was  only  somewhat  responsive. 


Also,  Hughes  did  not  provide  a  mini-program  of  instruction  and  specific  3-D 
training  modules  as  required. 

The  results  showed  that  effectiveness  of  coordination  was  very  good  in 
AMTESS  Phase  I,  but  coordination  appeared  to  be  less  effective  during  subse¬ 
quent  stages  of  the  project  due  to  increases  in  the  number  of  agencies  in¬ 
volved,  personnel  turbulence,  unanticipated  delays,  and  shortcomings  in  co- 
o*'dination  procedures. 

AMTESS  Phase  I  demonstrated  that  the  concept  of  generic  maintenance 
training  simulators  is  sound,  and  detailed  device  designs  were  produced. 
However,  in  terms  of  preparation  for  subsequent  development,  it  was  found 
that  only  a  general  plan  for  device  acceptance  existed,  which  did  not  provide 
sufficient  guidance  for  in-plant  inspections  of  the  devices.  Also,  there  was 
no  detailed  evaluation  plan  for  determining  the  training  effectiveness  of  the 
devices  agreed  upon  by  the  government. 

Utilization  of  Findings: 

Part  II  of  this  report  presents  guidelines  within  each  of  seven  types 
of  FEA  procedures,  some  of  which  were  conducted  in  AMTESS  and  some  of  which 
were  not  performed  in  AMTESS.  These  guide'  .nes  are  not  presented  as  step- 
by-step  procedures  for  use  in  training  device  development  efforts.  Rather, 
they  provide  useful  guidance  to  a  range  of  FEA  activities,  which  may  be  ap¬ 
plicable  to  future  development  efforts  such  as  AMTESS. 
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A.  Introduction 


The  Army  Maintenance  Training  and  Evaluation  Simulation  System 
(AMTESS)  represents  an  attempt  to  define  new  concents  in  the  area  of 
maintenance  training  (Dybas,  1981;  198'5;  Evans  &  Mirabel  la,  1982; 

Hofer,  1Q81).  Traditionally,  Army  students  receive  maintenance 
training  on  operational  eguipment.  However,  this  type  of  training  may 
be  counterproductive  for  several  reasons: 

•  Eguipment  may  be  damaged  during  training. 

•  Students  may  be  injured  during  operation  of  hazardous 
equ i pment . 

•  Availability  of  operational  equipment  for  training  purposes  is 
often  I imited. 

•  Actual  equipment  trainers  (AET)  may  be  less  cost  effective  than 
simulated  maintenance  equipment. 

As  an  effort  to  investigate  the  entire  process  of  specifying, 
designing,  and  procuring  training  devices,  the  AMTESS  program  had 
several  objectives.  One  of  the  Army’s  objectives  for  the  AMTESS 
program  was  to  develop  a  family  of  simulators  for  maintenance  training 
so  that  some  of  the  problems  which  occur  with  operational  equipment 
training  would  be  relieved.  Another  objective  was  to  develop  generic 
trainers  which  could  be  used  to  train  and  evaluate  maintenance 
procedures  and  tasks  for  all  major  maintenance  functional  areas. 
Other  objectives  were  to  develop  cost-effective,  training-effective, 
generic  maintenance  trainers  which  would: 

•  Support  introductory  training  in  a  school  setting 

•  Support  proficiency  training 

•  Provide  self-paced,  flexible  training 

•  Provide  feedback 

•  Record  and  display  performance  data 


•  Provide  for  automatic  administration  of  skill  qualification 
tests  (SOTs) 

•  Adapt  to  a  ranqe  of  Army  needs 

•  Use  recent  advancements  in  microcomputers,  video  storage,  and 
interactive  graphics 

The  AMTESS  project  was  sponsored  by  the  Army's  Project  Manager, 
Training  Devices  (PM  TRADE).  PM  TRADE'S  AMTESS  developmental  program 
consisted  of  two  phases.  During  the  first  phase,  new  training  device 
concepts  were  developed.  During  the  second  phase,  prototype  devices 
were  constructed  and  evaluated.  As  a  part  of  Phase  II,  the  U.S.  Army 
Research  Institute  (ARI)  was  tasked  to  conduct  an  evaluation  of  the 
prototype  models.  This  evaluation  was  undertaken  .as  part  of  a  larger 
effort  termed  SIMTRAIN.  This  was  accomplished  by  a  contractor  team 
originally  consisting  of  Honeywell,  Inc.  and  Science  Applications, 
Inc.  (SAI),  and  later  by  SAI  alone  (Evans  4  Mirabel  la,  1982;  Unger, 
Swezey,  Hays,  and  Mirabel  la,  1R84) 

The  Army's  first  SIMTRAIN  project  (termed  SIMTRAIN  I)  centered 
around  issues  involved  in  the  acquisition  of  training  devices.  More 
specifically,  SIMTRAIN  I  had  the  following  objectives: 

•  Develop  better  methods  for  determining  training  device 
requirements  and  characteristics  (Kane  &  Holman,  1982), 

•  Establish  procedures  to  Integrate  training  devices  into 
training  systems  and  develop  procedures  for  determining 
simulation  fidelity  requirements  (Baum,  Riedel,  Hays  & 
Mirabel  la,  1982) ,  and 

•  Perform  an  empirical  AMTESS  transfer-of-traini ng  evaluation 
(Unger,  Swezey,  Hays,  and  Mirabel  la  1984). 

An  expanded  effort,  termed  SIMTRAIN  II,  continues  the 
evaluation  and  includes  systematic  extens ions •  of  previous  AMTESS 
experiments.  The  present  report  Is  the  second  report  of  the  SIMTRAIN 


I!  projec+.  The  firs+  report  (Criswell,  Unger,  Swezey,  and  Hays, 
1985)  recounted  the  history  of  AMTESS  from  the  first  RFP  to  evaluation 
of  the  training  devices  developed  as  part  of  the  project. 


The  Criswell,  Unger,  Swezey  and  Hays  (1985)  report  was  broad  In 
scope  tracing  the  entire  histor.y  of  the  AMTESS  experience  and 
describing  the  role  and  evaluation  of  the  performance  of  participating 
organ i zatlons  including  government  agencies,  device  contractors,  and 
evaluation  contractors.  The  present  report  is  more  focused  in  scope. 
The  purpose  of  this  report  is  to  examine  the  front-end  analysis  (FEA) 
procedures  performed  by  the  government  agencies  and  contractors 
Involved  in  the  AMTESS  effort  and  to  measure  their  impact  on  the 
performance  and  utility  of  the  AMTESS  training  devices. 


B.  Purpose  of  This  Report 

AMTESS  represents  a  departure  from  tradition's!  training  device 
procurements,  since  AMTESS  was  an  experimental  research  and 
development  effort  designed  to  develop  new  Insights  \into  the  process 
of  device  acquisition.  While  front-end  analysis  is  important  in  all 
new  training  efforts,  it  is  particularly  critical  ror  the  AMTESS 
situation,  since  this  is  such  a  new  training  concept. ^The  AMTESS 
training  devices  are  generic  training  devices  in  which  combine 

two-dimensional  media  (e.g.,  CRT  monitors,  video-disc  projectors)  with 

J)  J? 

three  dimensional  ^plug-in”  mock-ups  of  actual  equipment  driven  by  a 
common  microprocessor.  ^ 


The  purpose  of  this  report  is  to  examine  the  front-end  analysis 
procedures  performed  for  AMTESS.  The  report  has  the  following  major 
objectives: 


•  Review  and  evaluate  FEA  procedures  performed  in  the  AMTESS 
efforts  as  a  basis  for  the  second  objective 

•  Develop  general  guidelines  for  FEA  procedures  for  future' 
training  device  development  procurements 
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This  report  is  not  confined  to  traditional  ^EAs  such  as  task 
analysis  and  training  requirements  analysis.  Rather,  the  report 
considers  a  variety  of  activities  performed  both  by  the  government 
agencies  and  the  contractors  involved  in  the  AMTESS  project  which 
helped  prepare  for  the  construction  and  evaluation  of  AMTESS  prototype 
devices.  The  timeframe  for  this  report  spans  the  period  from  Spring 
1977  when  a  feasibility  study  about  AMTESS  was  conducted  to  the  Summer 
of  19R0  when  the  contracts  to  develop  the  devices  were  awarded. 

The  guidelines  presented  in  this  report  are  intended  to  be 
broad  guides  for  future  front-end  analysis  in  AMTESS- 1  ike  simulator 
development  efforts.  The  guidelines  are  not  intended  to  be  detailed 
procedures  for  carrying  out  specific  front-end  analysis  procedures 
such  as  task  analysis.  Other  efforts  have  addressed  such  issues  Csee 
for  example  Demaree  (1961),  Miller  (1973),  Smith  (1965),  and  Wheaton, 
Rose,  Fingerman,  Korotkin,  Holding,  and  Mirabel  la  (1976)^.  Moreover, 
numerous  guidelines  already  exist  such  as  the  Army's  Instructional 
Systems  Development  (ISO)  model  (U.S.  Department  of  the  Army,  1975) 
and  the  Training  Device  Requirements  Documents  Guide  (PM  TRADE  and 
Army  Training  Support  Center,  1979).  Also,  the  Sev i  I  I e-Burtek  Phase 
II  reports  (Garlichs,  Miller,  and  Davis,  1983a, b,c)  provide  a  detailed 
description  of  their  AMTESS  specific  front-end  analysis  activities. 
The  strategy  of  the  present  report  was  to  develop  some  general,  easy 
to  understand  "do's"  and  "don'ts"  based  upon  a  case  study  of  the 
AMTESS  FEA's.  While  this  report  is  specifically  confined  to  the 
AMTESS  experience,  many  issues  raised  here  may  also  be  appropriate  for 
other  types  of  projects  designed  to  develop  training  system:;. 

The  remainder  of  this  report  contains  the  following  two  parts; 

Part  I 


•  Overview  of  relevant  FEA  literature 

•  Procedures  employed  in  the  present  effort 

•  Results 

•  Discussion 


Part  I  I 


•  FEA  Gu i de I i nes 

C.  Overview  of  Front-End  Analysis 

The  purpose  of  this  section  is  to  provide  a  brief  overview  of 
front-end  analysis  (FEA)  procedures  and  their  role  in  training  device 
development.  This  overview  provides  the  context  with  which  to  review 
the  AMTESS  project. 

Definition  of  FEA 


A  recent  workshop  of  participants  selected 
organizational  levels  from  all  of  the-  military 
appropriate  civilian  organizations  (Seidel  and  Wagner, 
the  following  definition  of  FEA: 


from  several 
serv ices  and 
1980)  produced 


Front-end  analysis  (FEA)  Ic  a  process  that  evaluates 
requirements  for  manpower,  personnel  and  training  (MP&T)  during 
the  early  stages  of  the  military  systems  acquisition  cycle.  Its 
purpose  is  to  (1)  determine  manpower,  personnel  and  training 
requirements  under  alternative  system  concepts  and  designs,  and 
(2)  estimate  the  impact  of  these  MP&T  requirements  on  system 
effectiveness  and  life  cycle  costs.  Its  end-product  should  be 
the  information  needed  to  assume  that  effective  resources 
(human,  equipment,  materiel)  will  be  available  when  ard  as 
required  for  each  system  to  achieve  its  intended  contribution 
to  military  readiness  and  effectiveness  (p.  1). 


The  results  of  this  workshop  also  noted  that  recent  policy 
initiatives  (0MB  Circular  A-109,  OoD  Directives  5000.1  and  5000.2)  are 

intended  to  force  the  timely  application  of  FEA  procedures  within  the 
overall  materiel  acquisition  process  with  particular  attention  to  life 
cycle  support. 


I SD  Approach  to  FEA 


Training  establishments  of  all  military  services,  including  the 
Army,  have  adopted  the  Instructional  Systems  Development  (ISD)  model 
as  an  overall  approach  to  design,  development,  implementation,  and 
control  of  virtually  all  military  training.  ISD  is  intended  to  be  a 
comnrehens : ve  and  systematic  approach  to  training.  The  ISD  model 
involves  an  interlocking  series  of  five  phases  (U.S.  Department  of  the 
Army,  1975).  These  phases  are: 

•  Analyze 

•  Design 

•  Develop 

«  Implement 

•  Control 

At  a  molar  level,  this  model  is  eminently  straightforward  in 
concept  and  intent.  One  of  the  key  features  of  the  ISD  approach  is  to 
place  appropriate  emphasis  upon  systematic  analysis  of  the  entire  job 
performance  situation  of  interest  in  order  to  assure  that  all  training 
solutions  are  focused  on  we  1  I -documented  performance  needs. 

Within  the  ISD  framework,  front-end  analysis  is  a  term 
referring  to  Job  and  task  analysis,  selection  of  tasks  for  training, 
and  development  of  job  performance  measures  (U.S.  Department  of  the 
Army,  1975).  (?ener ical  ly,  the  term  FEA,  is  applied  to  a  wide  variety 
of  purposes^  activities,  and  products,  all  of  which  ostensibly  occur 
early  in  the  I SD  •  process.-  These  FEA  activities  are  critically 
important  to  the  eventual  outcome  of  applying  the  ISD  model  since  "if 
a  performance  requirement  is  overdefined  or  underdefined  based  on 
analyses  of  field  conditions,  a  new  weapons  system,  or  other  source, 
the  ISD  procedures  will  tend  to  exaggerate  the  error"  (U.S.  Department 
of  the  Army,  1975,  p.  2). 

A  variety  of  types  of  FEA  are  available  and  are  conducted,  as 
appropriate,  in  selected  train*  ituations.  Among  the  types  of  FEA 


are  the  fol lowi ng; 


•  Task  analysis 

•  Job  analysis 

•  Training  requirements  analysis 

•  Fidelity  anal ys i s 

•  Human  engineering  analysis 

•  Organization  analysis 

•  Person  analysis 

•  Re  I  i  ab i I i ty-Ava i i ab i I ity-Mai  ntai nab  i  I  i  ty  analysis 

•  Safety  engineering  analysis 

•  Cost  effectiveness  analysis 

•  Training  effectiveness  analysis 

The  last  phase  of  the  ISO  model  is  referred  to  as  Control.  The 
Control  phase  includes  both  an  internal  evaluation  component  and  an 
external  evaluation  component.  The  Internal  evaluation  component  Is 
Intended  to  provide  the  basis  for  determining  whether  the  training 
developed  as  a  consequence  of  the  FEA  activities  achieved  its 
objecTives.  The  external  evaluation  component  is  intended  to  assure 
that  those  who  complete  the  training  meet  the  requirements  of  the  job 
situation  as  determined  by  the  findings  from  the  FEA. 

In  particular,  planning  for  both  internal  and  external 
evaluation  must  begin  at  the  earliest  stages  of  the  ISO  approach  in 
concert  with  the  FEA  activities.  For  Instance,  internal  evaluation 
requires  input' from  each  of  the  preceding  phases  of  the  ISO  approach, 
and  the  external  evaluation  requires  the  internal  evaluation  results 
as  well  as  the  FEA  results.  Thus,  although  evaluation  nominally  marks 
the  final  phase  of  the  I SD  approach,  it  is  linked  directly  to  the 
initial  FEA  activities. 

Reviews  of  FEA  Procedures 


Wheatcn,  Pose,  Flngerman,  Korotkin,  Holding  and  Mirabel  la 
(1976)  reviewed  the  diverse  and  somewhat  fragmented  literature  on 


+raininq  device  effectiveness  models,  including  the  FEA  procedures 
involved  in  these  various  approaches.  They  found  that  most  models  were 
prescriptive  of  training  device  design  based  on  the  FEA  procedures 
employed.  For  instance,  Miller  (1954)  began  by  breaking  down  an 
operational  situation  into  missions,  which  were  then  analyzed  to 
enumerate  tasks  with  respect  to  their  chronological  sequence  and 
similarity  of  content.  These  tasks  were  then  arranged  along  one 
dimension  of  a  matrix  and  training  strategies  along  the  other 
dimension.  Another  model  which  is  based  on  an  initial  task  analysis 
is  that  offered  by  Shettel  and  Horner  (1972),  which  identified 
significant  learning  elements  in  terms  of  behavioral  categories  and 
then  identified  training  techniques  related  to  each  behavioral 
category.  Although  both  of  these  methods  offer  a  systematic  approach 
to  FEA  procedures,  they  rely  upon  ambiguous  definitions  of  task 
detail,  and  the  basic  models  place  heavy  reliance  on  expert  judgment. 

Apart  from  the  task-analytic  approaches  noted  above,  Caro 

(1970)  has  presented  a  method  which  Is  designed  to  identify  the 
commonality  between  the  stimulus-response  conditions  in  a  training 
device  and  the  stimulus-response  cpnditions  in  an  operational 
situation.  This  approach  requires  a  detailed  analysis  of  these 

stimulus-response  connections  followed  by  a  comparison  of  the  rated 
degree  of  fidelity  between  the  training  device  and  the  operational 
situation.  Another  stimulus-response  based  model  is  offered  by  Altman 

(1970)  from  which  can  be  derived  an  index  of  net  transfer  (from 

training  to  operational  situation).  This  net  transfer  Index  is  an 
algorithmic  summation  of  the  various  probabilities  associated  with  the 
stimulus  and  response  elements  in  both  the  training  and  operational 
situations. 

After  reviewing  these  and  other  models,  Wheaton,  Rose, 
Fingerman,  Korotkin,  Holding,  and  Mirabel  la  (1976)  concluded  that: 

•  None  of  the  models  adequately  considers  both  acqu.isition  and 

transfer  aspects. 


•  Definitions  and  procedures  for  data  collection  and  processing 
tend  to  be  complex  and  ambiguous. 

•  There  is  a  tendency  to  oversimplify  the  problem,  both  in  terms 
of  the  numbers  of  pertinent  factors  and  the  level  of  analysis. 

In  addition,  their  efforts  to  address  these  issues  resulted  in 
TRAINVICE,  a  training  system  model  predictive  of  transfer  of  training 
based  upon  task,  device,  learning  deficit,  and  instructional  feature 
variables  (Tufano  and  Evans,  in  press;  Knerr,  Nadler,  Dowell  and 
Tufano,  198^;  Swezey,  1983;  Swezey  and  Evans,  1980).  Also  developed 
were  scales  for  measuring  these  variables  and  a  formula  for  combining 
the  measures  into  an  index  value  of  transfer  of  training.  Other 
examples  of  of  work  addressing  these  issues  are  the  efforts  of  Schulz 
and  others  (Schulz,  1979;  Schulz,  Underhill  and  Hargan,  1979;  Schulz, 
Underhill,  Hargan  and  Wagner,  1979;  Schulz  and  Farrell, 
1980a, b,c,d,e,f;  Schulz  and  Wagner,  1981). 

Air  Force  FEA  Procedures 

The  Air  Force  has  implemented  the  ISD  process  through  its 
Handbook  for  Designers  of  Instructional  Systems  (U.S.  Department  of 
the  Air  Force,  1978).  This  handbook  is  a  detailed,  procedura I i zed 
guide  on  how  to  implement  the  ISD  model  with  particular  emphasis  upon 
task  analysis  and  setting  criterion  objectives  since  these  two  topics 
often  present  difficulty.  With  respect  to  task  analysis,  this 
handbook  breaks  this  process  into  two  principal  steps: 

•  Identifying  job  performance  requirements 

•  Identifying  training  requirements 

Job  performance  requirements  describe  what  people  must  do  to 
perform  their  jobs.  This  step  is  accompiished  in  the  following 
systematic  manner. 


■.currwj LULi^^Lwj^JU i-F J<^  I  j tWi  W J!J1  W i?lil-.%  L^J 


•  r,ollect  information 

•  Identify  duties,  tasks,  and  subtasks 

•  Determine  which  tasks  require  instruction 

•  Describe  tasks  and  their  activities 

Once  the  job  performance  requirements  have  been  identified,  the 
nex+  step  is  to  identify  training  requirements  which  are  specified  in 
terms  of  the  knowledges  and  skills  required  to  satisfy  the  Job 
performance  requirements.  These  training  requirements  represent  new 
knowledges,  skills,  and  proficiency  levels  which  trainees  must  acquire 
in  order  to  achieve  the  job  performance  requirements.  Analysis  then 
identifies  those  training  requirements  which  require  practice  and 
therefore  represent  training  standards. 

.  In  general,  the  Air  Force  ISD  process  begins  at  a  very  global 
level  (l.e.,  the  job  and  how  it  fits  within  a  broad  mission)  and  it  is 
successively  refined  in  greater  detail  to  the  specification  of 
training  requirements  and  associated  training  standards.  Underlying 
this  entire  process  is  the  orderly  compilation  and  development  of 
Information  about  the  job  and  its  components.  Included  among  the 
sources  for  this  information  are: 


•  Statements  of  system  requirements  and  functions 

•  Listings  of  duties,  tasks,  a-.d  subtai.V'.s 

•  Task  data 

•  Descriptions  of  task  activities  and  performance  standards 

•  Listings  of  skills  and  knowledges 

•  Resource  and  personnel  data 


One  of  the  major  differences  between  the  FEA  procedures  used  by 
the  Air  Force  relative  to  the  other  services  is  a  greater  emphasis  on 
occupational  surveys  of  job  Incumbents  for  identifying  the  tasks 
included  in  a  job  as  well  as  indicators  of  activity  distribution  and 
criticality  of  these  tasks. 
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The  Air  ^orce  has  conducted  a  great  deal  of  work  in  the  area  of 
maintenance  training  simulators.  In  common  with  the  other  military 
services,  Air  Force  policy  requires  application  of  ISO  procedures 
(including  FEA)  to  development  and  procurement  of  maintenance  training 
simulators  (Purifoy  and  Benson,  1979).  Although  the  Air  Force 
prescribes  the  application  of  FEA  procedures  to  the  development  of 
training  simulators,  several  problems  have  been  identified.  For 
example,  the  Air  Force  has  found  that,  in  practice,  there  is  a  lack  of 
both  procedural  guidance  In  performing  FEA  as  well  as  a  lack  of 
detailed  task  analysis  data,  particularly  for  existing  weapon  systems. 

In  addition,  there  is  a  tendency  to  develop  major  overall  simulators 
rather  than  part-task,  or  lower  fidelity  devices  regardless  of  the  FEA 
results  or  general  learning  principles.  With  respect  to  evaluation, 
Air  Force  procedures  make  it  a  part  of  the  acquisition  process,  but 
again,  little,  if  any,  systematic  guidance  is  provided  on  how  the 
evaluation  activities  are  linked  to  the  FEA  activities. 

Navy  FEA  Approaches 

The  Navy  has  also  implemented  the  ISO  model.  Including  FEA 
procedures,  in  a  manner  very  similar  to  the  other  military  services 
(M I L-T-29053,  1977),  The  Navy's  approach  include?*  somewhat  more 

reliance  upon  subject  matter  experts  (SME)  as  a  principal  source  of 
job  and  task  data  than  do  the  other  services. 

Additional  Sources  of  FEA  Procedures 

A  recent  compendium  of  procedures,  methods,  and  tools 
applicable  to  FEA  represents  a  reasonably  complete  collection  of  - 
references  on  FEA  throughout  the  weapon  system  acquisition  process 
(Dynamics  Research  Corporation,  1982).  A  total  of  444  references  ; 
reviewed  within  the  following  12  categories. 

•  Acquisition  Process 

•  Cost 

•  Equipment 


•  Front-End  Analysis 

•  Human  Factors 

•  Logistics 

•  Manpower 

•  Manpower,  Personnel,  and  Training. 

•  Personnel 

•  Software 

•  Training 

•  Data  Bases 

As  noted,  this  compendium  was  assembled  against  the  backdrop  of 
the  overall  weapon  system  acquisition  process. 

As  documented  by  the  references  in  this  compendium,  there  are  a 
variety  of  methods  and  procedures  which  can  be  applied  in  the 
execution  of  FEA.  For  instance,  there  are  automated  procedures  and 
non-automated  procedures,  and  some  procedures  are  highly  structured 
while  others  are  less  structured.  Also,  ARI  is  involved  in  an  ongoing 
effort  designed  to  develop  an  automated  procedure  for  extracting 
information  from  SME's  to  aid  training  designers  and  course  developers 
in  making  decisions  about  what  to  train,  where  to  train,  and  how  to 
train.  In  general,  however,  these  various  procedures  are  intended  to 
determine  the  specific  need  for  training,  identify  the  task/skill 
areas  to  be  trained,  select  from  alternative  methods  of  instruction, 
and  make  a  preliminary  assessment  of  the  cost  and  training 
effectiveness  of  the  alternative  chosen. 

Of  more  direct  relevance  to  the  current  report  is  the  Training 
Device  Requirements  Documents  Guide  (PM  TRADE  and  Army  Training 
Support  Center,  1979).  Major  portions  of  the  TRAINVICE  model  noted 
above  were  Implemented  in  this  guide.  The  purpose  of  this  guide  is  to 
present  the  structured  procedures  for  responsible  project  officers 
within  TRADX  schools  who  prepare  training  device  requirements 
documents.  Included  among  these  procedures  are  FEA  activities.  The 
FEA  activities  documented  In  this  Guide  are  concerned  with  evaluating 
a  training  need  to  determine  if  it  is  a  training  requirement,  and 


specifying  these  training  requirements  in  terms  of  tasks,  skills,  and 
objectives.  Following  specification  of  training  reau i rements ,  a  media 
selection  analysis  is  performed  to  determine  if  a  training  device 
requirement  is  appropriate. 

After  identifying  the  general  requirements  for  a  training 
device,  the  following  analyses  are  conducted  to  identify  what  kind  of 
training  device  is  required: 

•  task  commonality  analysis 

•  physical  similarity  analysis 

•  functional  similarity  analysis 

•  skill  and  knowledge  requirements 

•  task  training  difficulty  analysis 

•  training  device  effectiveness  analysis 

•  cost  analysis 

The  outputs  of  these  various  analyses  then  are  evaluated  in 
greater  detail  to  determine  which  of  the  following  requirements 
documents  are  appropriate: 

•  Training  Device  Letter  of  Agreement  (TDLOA)  or  a  Commercially 
Available/Fabricated  Training  Device  Requirement  (CAFIDR) 
document 

•  Training  Device  Requirement  (TDR)  or  a  Training  Device  Letter 
Requirement  (TDLR) 

The  choice  between  each  of  these  pairs  of  requirements 
documents  is  based  on  specified  cost  and  scheduling  parameters,  and 
the  various  choices  result  in  different  levels  of  approval  authority 
and  budgetary  authorizations.  These  decisions  also  establish  the 

broad  decision  criteria  for  whether  a  Joint  Working  Group  should  be 
formed  to  articulate  issues  and  outline  developmental  strategies. 
Essentially,  the  procedures  in  the  Training  Requirements  Documents 
Guide  are  directed  at  the  formulation  of  training  device  requirements 


documents  with  minims  I  reference  to  actual  development  and  eventual 
evaluation  of  the  device  itself. 

Among  the  cone i us  ions  reached  by  the  various  participants  in 
the  invitationai  workshop  noted  previousiy  (p.  6)  were  the  foi lowing 
(Seidei  and  Wagner,  19fi0): 

•  There  are  major  differences  among  the  service  efforts  in 
maturity  of  effort,  level  of  detail,  and  applicability. 

•  There  is  a  substantial  need  for  standardized  definitions  of 
such  terms  as  task  analysis,  measures  of  effectiveness, 
measures  of  cost,  and  transfer  of  training. 

•  The  services  have  substantial  information  about  FEA  which 
should  be  compiled  and  made  more  available. 

•  No  FEA's  have  been  validated. 

•  Managerial  and  institutional  aspects  of  FEA  deserve  more 
attention  to  include  recognizing  and  strengthening  the 
incentives  for  program  managers  to  Identify  roles  for  R&D 
managers  and  training  developers. 

In  summary,  although  there  is  a  relative  abundance  of 
information  on  the  topic  of  FEA,  the  area  nonetheless  is  in  some 
disarray.  There  I«  basic  agreement  on  the  need  for  several  principal 
FEA  components  such  as  task  analysis,  training  requirements  analysis, 
and  fidelity  analysis  (in  the  case  of  training  devices).  However, 
there  exists  a  variety  of  methods  for  performing  these  activities. 
These  methods  often  vary  in  the  level  of  detail  and  systematization 
required  for  their  execution.  More  Importantly,  FEA  activities 
typically  are  focused  primarily  on  the  design  and  development  of 
training  rather  than  on  providing  a  clear  linkage  to  evaluation 
activities.  Portions  of  the  SIMTRAIN  I  project  were  designed  to 
address  this  absence  of  procedural  guidance  (Eberts,  Smith,  Dray  and 
Vestewig,  1982),  and  ARI  research  to  close  this  gap  further  is 
underway. 


This  section  has  presented  a  brief  overview  of  FEA  procedures 
from  a  theoretical  as  well  as  operational  perspective  in  order  to  show 
the  imoortance  of  their  role  in  system  acquisition  and  traininq  device 
development.  In  subsequent  sections  of  this  report,  the  FEA 
procedures  which  were  performed  in  the  AMTESS  project  are  reviewed  and 
d i scussed . 

D.  Procedures 

Since  this  report  addresses  AMTESS  FEA  activities  which 
occurred  as  many  as  6  years  aqo,  the  principal  method  employed  was  a 
review  of  existing  AMTESS  documentation.  This  section  describes  the 
documents  which  were  reviewed  for  this  task  and  the  procedures 
employed  for  analyzing  this  information. 

1.  Documents  Reviewed 

Three  types  of  documents  were  reviewed  for  this  report.  These 

were: 

•  Relevant  Army  doctrine  and  related  government  documentation 

•  AMTESS  requirements  documents  and  reports 

•  SIMTRAIN  requirements  documents  and  reports 

Each  of  these  sources  is  discussed  in  more  detail  below. 

a.  Government  Documentation.  Although  this  report  deals 
exclusively  with  FEA  within  the  context  of  the  AMTESS  development 
process,  the  overall  AMTESS  initiative  exists  within  the  broader 
context  of  military  training  procedures  (e.g.,  U.S.  Department  of  the 
Army,  1975)  and  training  device  development  (e.g.,  PM  TRADE  and  Army 
Training  Support  Center,  1979).  Thus,  existing  documentation  regarding 
Army  doctrine  applied  to  training  device  development  procedures  was 
reviewed.  Also,  this  sou^'ce  of  information  included  related  studies 
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and  analyses  on  the  topic  of  training  device  development  (Dynamics 
Research  Corporation,  1982). 

b.  AMTESS  Documents.  The  second  source  of  information 
included  relevant  documentation  from  the  AMTESS  initiative  itself.  For 
example,  the  Request  for  Procurement  (RFP)  and  associated  Statements 
of  Work  (SOW)  for  the  AMTESS  I  effort  was  included  In  the  review.  This 
category  also  included  the  complete  final  reports  of  the  four  AMTESS  I 
contractors  (i.e.,  Grumman  Aerospace  Corporation,  Honeywell  Systems 
and  Research  Center,  Hughes  Aircraft  Company,  and  a  consortium 
consisting  of  Seville  Research  Corporation  and  Burtek,  Inc.).  Other 
relevant  documentation  items  included  in  this  category  were  the 
recorded  minutes  of  advisory  group  meetings  as  well  as  notes  and 
working  file  materials  from  individuals  involved  in  the  AMTESS  effort. 
(These  latter  materials,  due  to  their  informal  nature,  were  used 
primarily  for  background  information). 

c.  SIMTRAIN  Documents.  The  third  category  of  information 
included  the  following  two  documents: 

•  Draft  final  report  of  the  SIMTRAIN  I  tr ansfer-of-tr a i n i ng 

evaluation  effort  (Unger,  Swezey,  Hays,  and  Mirabel  la,  1984) 

•  Task  1  report  of  the  SIMTRAIN  II  effort.  (Criswell,  Unger, 

Swezey,  and  Hays,  1983) 

The  second  of  these  documents,  the  SIMTRAIN  II  Task  1  report, 
is  based  on  a  series  of  interviews  with  various  individuals  and 
organizations  involved  in  the  entire  AMTESS  effort,  and  as  such, 
represents  the  only  primary  informatior.  for  the  present  study. 

All  documents  reviewed  In  the  present  study  are  listed  in  the 
References  section  of  th's  report. 
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2.  Analysis  Procedures 


In  reviewing  these  materials,  the  major  focus  was  upon  the  FEA 
activities  prescribed  by  the  AMTESS  I  contract  and  conducted  by  the 
contractors  (i.e.,  Grumman,  Honeywell,  Hughes,  and  Sevi I le-Burtek)  . 
Given  that  FEA  procedures  are  a  component  of  the  entire  ISO  process 
(and,  therefore,  the  training  device  development  process),  this  review 
extends  beyond  FEA  activities  themselves  which  were  performed 
primarily  by  the  contractors  to  include  activities  performed  by  the 
government. 

The  criteria  chosen  to  guide  this  review  were: 

a.  Clarity  of  Requirements.  This  criterion  is  applied 
exclusively  for  reviewing  the  performance  of  the  government  in  laying 
out  the  requirements  for  AMTESS  I  in  the  AMTESS  I  RFP.  Clarity  of 
requirements  deals  with  the  extent  to  which  the  RFP  provided 
sufficient  and  appropriate  guidance  based  on  the  deliverables  produced 
at  the  completion  of  AMTESS  I. 

b.  Completeness  of  Performance.  This  criterion  is  applied 
primarily  for  reviewing  the  performance  of  the  AMTESS  device 
development  contractors  since  their  performance  was  designed  to 
fulfill  the  terms  of  their  contract.  The  standard  applied  to 
completeness  of  performance  in  this  context  is  the  extent  of 
compliance  with  contractual  requirements. 

c.  Effectiveness  of  Coordination.  Given  the  scope  and 
implications  of  the  AMTESS  initiative,  there  are  a  number  of  separate 
organizations  which  are,  or  have  been,  involved  in  AMTESS,  including 
the  FEA  activities.  Therefore,  the  third  criterion  addressed  matters 
of  coordination  among  these  organizations. 

d.  Preparation  for  AMTESS  ll/SIMTRAIN  I.  The  main  purpose  of 
AMTESS  I  was  to  develop  device  concepts  for  the  development  of  the 
prototype  training  devices  (AMTESS  II)  and  their  evaluation  (SIMTRAIN 


I).  Since  at  the  end  of  the  AMTESS  I  effort  only  a  portion  of  the 
overall  developmental  process  had  been  completed,  it  is  therefore 
necessary  to  examine  AMTESS  I  within  the  broader  context  of  both 
AMTESS  I  and  II.  Essentially,  this  means  assessing  the  relationship 
between  what  was  done  in  AMTESS  II  and  what  was  produced  in  AMTESS  I, 

E.  Results 


This  section  describes  various  front-end  analytic  activities 
implemented  prior  to  and  during  the  AMTESS  I  project.  As  noted 
earlier,  the  scope  of  this  effort  has  been  defined  broadly  to  include 
various  aspects  of  FEA  beyond  task  analysis  and  training  requirements 
analysis.  The  events  covered  in  this  section  encompass  the  period 
from  May  1977  (the  initial  study  advisory  group  (SAG)  for  maintenance 
training  meeting)  and  July,  1980  (the  delivery  of  AMTESS  I  final 
reports  by  the  contractors). 

The  section  begins  with  a  brief  history  of  the  activities 
during  this  period  (see  Criswell,  Unger,  Swt?zey  and  Hays,  1983  for 
more  detail).  It  then  provides  a  discussion  cf  key  documents  produced 
during  the  period. 


History  of  AMTESS  Front-End  Activities 


This  section  outlines  events  which  took  place  prior  to  the 
building  of  the  prototype  AMTESS  training  devices,  and  which  set  the 
stage  for  the  construction  of  these  devices.  Table  1  contains  a 
summary  of  these  events. 


Government  activities  leading  up  to  the  procurement  for  the 
prototype  devices  can  be  traced  back  at  least  as  far  as  the  Spring  of 
1977  when  an  RFP  was  issued  to  conduct  a  study  of  the  feasibility  of 
using  reduced  physical  fidelity  trainers  for  maintenance  training.  In 
May  of  1977  the  initial  Maintenance  Training  Study  Advisory  Group 
(SAG)  -meeting  was  held  to  review  the  proposals  received  in  response  to 
the  RFP.  The  SAG  consisted  of  members  of  PM  TRADE  and  the  U.S.  Army 
Training  Support  Center  (USATSC). 


TABLE  1 


A  contract  was  awarded  to  the  Seville  Research  Corporation  to 
develop  alternative  conceptual  designs  of  reduced  physical  fidelity 
trainers  that  might  be  cost  effectively  substituted  for  actual 
equipment  trainers.  During  the  course  of  Sevil'e's  effort,  periodic 
SAG  meetings  were  held  to  review  findings.  During  this  period,  the 
SAG  was  expanded  to  include  both  the  original  two  agencies,  PM  TRADE 
and  LISATSC,  and  representatives  from  the  Armor  School  and  Ordnance 
Center. 

In  May  of  1978,  USATSC  sponsored  a  maintenance  training 
requirements  conference  at  Ft  Lee,  VA.  The  conference  activities 
included  a  discussion  of  reduced  physical  fidelity  trainers  as 
substitutes  for  Actual  Equipment  Trainers  ( AET  / .  There  were  two 
important  outcomes  of  this  meeting.  One  outcome  was  that  U.S.  Army 
Air  Defense  School  (USAADS)  and  U.S.  Army  Missile  Munitions  Center  and 
School  (USAMMCS)  developed  a  letter  requirement  for  maintenance 
trainers  for  an  Improved  Hawk  maintenance  trainer.  The  letter 
contained  a  statement  of  maintenance  training  needs  and  the  principal 
characteristics  desired  in  a  modular  training  device  for  Improved  Hawk 
Equipment.  This  letter  became  an  input  for  the  AMTESS  I  RFP.  A 
second  outcome  was  that  PM  TRADE  was  announced  as  the  agency  to 
coordinate  and  manage  the  contract  designed  to  procure  the  initial 
reduced  physical  fidelity  trainers. 

In  July  of  1978,  PM  TRADE  invited  a  number  of  agencies  to  send 
representatives  to  a  Joint  Study  Advisory  Group  (JSAG)  to  assist  with 
the  procurement  for  the  training  devices.  An  initial  schedule 
released  at  this  time  indicated  that  the  prototype  devices  would  be 
delivered  by  September,  1980  with  subsequent  evaluation  of  these 
devices  completed  by  January,  1981. 

The  first  JSAG  meeting  was  held  in  August  of  1978.  The  major 
agenda  item  was  to  agree  upon  objectives  for  the  AMTESS  I  RFP.  The 
RFP  was  released  subsequently  in  March  of  1979.  Since  the  release 
date  of  the  RFP  was  several  months  behind  the  previously  scheduled 
release  date  of  October,  1978,  the  project  schedule  was  revised  to 
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reflect  prototype  delivery  in  April,  1981  and  completed  evaluation  by 
September,  1981.  It  should  be  noted  that  in  the  Fall  of  1978  the  JSAG 
was  retitled  the  Joint  Working  Group  ( JWG) . 

Prior  to  the  release  of  the  AMTESS  I  RFP  (January,  1979),  PM 
TRADE  had  circulated  a  Memorandum  of  Understanding  (MOU)  for  signature 
by  representi ves  of  the  fol lowing  agencies: 

PM  TRADE  -  Project  Manager  Training  Devices 

USAADS  -  U.S.  Army  Air  Defense  School 

USATSC  -  U.S.  Army  Training  Support  Center 

USMMCS  -  U.S.  Army  Missile  Munitions  Center  &  School 

USnc&S  -  U.S.  Army  Ordnance  Center  and  School 

The  MOU  was  intended  to  outline  the  respective  tasks  of  the 
JSAG  member  agencies  and  to  formalize  their  relationships.  fb  fully 
signed  copy  of  the  MOU  could  be  obtained,  therefore,  it  is  not  clear 
which  agencies  actually  signed  the  MOU. 

The  proposals  for  the  RFP  were  evaluated  in  June  of  1979  and 
the  contracts  began  in  September  with  oral  project  plans  presented  by 
the  four  winning  contractors,  Grumman  Systems  and  Research  Center, 
Honeywell  Aerospace  Corporation,  Hughes  Aircraft  Company,  and  a  team 
of  Seville  Research  Corporation  and  Burtek,  Inc..  At  this  time, 
another  revised  project  schedule  was  released.  This  schedule 
projected  prototype  delivery  in  July  of  1981,  with  evaluation  complete 
by  December  of  1981.  Subsequent  to  the  award  of  the  AMTESS  I 
contracts,  the  U.  S.  Army  Research  Institute  (ARI)  agreed  to 
participate  in  preparing  an  evaluation  plan  for  evaluating  the 
prototype  devices.  In  December  of  1979,  ARI  officially  accepted  a 
role  in  the  evaluation  of  the  devices  by  signing  the  MOU  although  ARI 
was  not  Involved  In  the  initial  planning  for  the  development  and 
evaluation  of  the  devices.  In  January  of  1980,  ARI  delivered  to  PM 
TRADE  their  proposed  evaluation  plan.  (This  plan  is  described  In  a 
subsequent  section  of  this  report). 


In  March  of  1980,  the  contractors  presented  oral  briefings  of 
the  results  of  the  AMTESS  I  effort.  By  July,  1980,  all  contractors 
had  submitted  their  final  reports,  thus  completing  the  front-end 
activities  leading  up  +o  the  development  of  the  prototype  simulators. 

The  next  part  of  this  section  provides  reviews  of  the  seven  key 
documents  produced  prior  to  AMTESS  II  along  with  a  summary  of  a 
critical  coordination  event,  the  "kick-off"  meeting  for  the  offical 
AMTESS  I  contractual  effort. 


Review  of  Key  Documents 


a.  Seville  Feasibility  Study.  Seville  was  contracted  by  PM 
TRADE  prior  to  the  preparation  of  the  AMTESS  I  RFP  to  conduct 
exploratory  research  concerning  reduced  physical  fidelity  trainers. 
According  to  the  final  report  for  that  project  (Dural  I,  Spears,  & 
Prophet,  1978)  the  effort  had  the  following  two  purposes: 


•  Develop  conceptual  descriptions  for  a  variety  of  reduced 
physical  fidelity  training  devices  fhat  might  be  cost 
effectively  substituted  for  AET,  and 


•  Develop  an  evaluation  methodology  for  determining  the  training 
effectiveness  of  the  conceptual  devices  should  they  be 
developed  and  provided  by  the  Army. 


The  first  task  Seville  undertook  in  the  effort  was  to 
investigate  the  use  of  AET's  in  maintenance  training  and  to  identify 
those  potential  AET  replacement  areas.  Based  on  these  findings, 
Seville  then  evaluated  existing  reduced  physical  fidelity  alternatives 
and  began  the  development  of  the  new  reduced  physical  fidelity 
concept. 


Seville's  examination  of  current  Army  maintenance  training  was 
confined  to  automotive,  track  vehicle,  and  heavy  equipment  maintenance 
training.  One  key  finding  in  this  area  was  that  there  was  an  overlap 


of  training  across  some  of  these  areas.  In  particular,  automotive 
(MOSs  6'^HIO  and  6'^H2n)  and  heavy  equipment  mechanics  (MOSs  62B10  and 
62B20)  receive  training  in  maintenance  of  both  wheeled  and  tracked 
vehicles.  Furthermore,  MOS  63C10  (track  vehicle  mechanics)  received 
training  on  wheeled  vehicles. 

Based  on  their  analysis,  Seville  recommended  four  areas  be 
addressed  in  examining  reduced  physical  fidelity  alternatives.  These 
four  areas  included  the  following: 

•  Troubleshooting  engines  and  related  systems 

•  Troubleshooting  track  vehicle  track/suspension  systems 

•  Remove  I /rep  I acement  of  engine  and  power  packs 

•  Troubleshooting  turret  electrical  and  hydraulic  systems 

The  first  three  of  these  areas  are  the  responsibility  of  wheel 
and  track  vehicle  maintenance  personnel  while  the  fourth  is  the 
responsibility  of  the  turret  mechanic. 

Seville  next  examined  existing  reduced  physical  fidelity 
alternatives  for  training  in  these  areas.  Surprisingly,  they 
uncovered  only  one  reduced  physical  fidelity  device  to  be  in  current 
use,  the  flat  panel  systems  simulator  (FPSS).  According  to  Seville 
"these  devices  are  used  primarily  in  teaching  system  troubleshooting 
tasks  in  maintenance  training  and  In  teaching  procedural  tasks  for 
operator  training  (Dural  I,  Spears,  and  Prophet,  V978,  p.19)."  Seville 
found  that  FPSSs  are  infrequently  used  for  teaching  motor  and 
cognitive  tasks  of  the  type  performed  In  the  equipment  maintenance 
tasks  they  had  reviewed. 

The  Seville  report  suggested  that  FPSS  would  have  some  utility 
for  AET  substitution  in  troubleshooting  engines  and  turret  electrical 
and  hydraulic  systems.  However,  they  saw  much  more  limited  utility 
for  tasks  such  as  engine  removal  or  troubleshooting  track/suspension 
systems . 


With  this  review  complete,  Seville  set  out  to  develop  new 
reduced  physical  fidelity  concepts.  The  strategy  of  the  reduced 
physical  fidelity  device  was  to  "bridge  the  gap"  between  AET  which  is 
the  primary  vehicle  for  hands-on  training  and  simulators  which 
primarily  train  diagnosis  and  troubleshooting.  The  reduced  physical 
fidelity  device  would  be  a  hybrid  containing  a  3-D  mock-up  of  the 
eguipment  ( for • tra i n i ng  motor  tasks)  a  FPSS  for  troubleshooting  (e.g., 
inspecting  gauges,  reading  meters),  and  a  microprocessor  for 
coordinating  all  instruction.  This  basic  system  would  have  the 
capability  of  training  many  maintenance  tasks,  although  each  of  the 
three  components  would  not  necessarily  be  reguired  for  training  each 
task . 


Seville  designed  a  straightforward  method  for  evaluating  the 
reduced  physical  fidelity  devices.  The  basic  strategy  was  to  evaluate 
the  reduced  physical  fidelity  device  relative  to  AET  and  relative  to 
alternative  reduced  physical  fidelity  devices.  The  design  was  a  three 
way  analysis  of  variance  which  included  training  device  (reduced 
physical  fidelity,  AET,  other),  mode  of  instruction  (lock-step, 
individually  paced),  and  trainee  characteristics  (2  to  4  levels).  The 
major  criteria  for  trainee  characteristics  were  aptitude  and 
experience.  Seville  felt  that  trainees  could  be  stratified  into  two 
to  four  levels  on  these  characteristics.  Seville  also  noted  that  a 
technigue  for  assigning  instructors  would  need  to  be  developed  and 
suggested  that  a  control  for  traini,.g  site  might  be  required. 

The  criterion  measures  for  evaluating  the  devices  fell  into  the 
following  three  categories: 

•  Specific  measures  taken  in  close  approximation  to  the  training 

•  Actual  performance  on  the  job 

•  User  acceptance  of  the  devices 

Seville’s  evaluation  plan  did  not  treat  sample  sizes  required 
or  amount  of  time  required,  nor  did  it  discuss  potential  problems 
related  to  testing  in  a  field  environment. 


b.  AMTESS  I  RFP.  The  AMTESS  I  RFP  (DAAK51-79-0-001 B)  laid  the 
foundation  for  the  prototype  devices  delivered  during  AMTESS  II.  The 
purpose  of  AMTESS  I  was  to  "conceptually  design  a  modular  maintenance 
trainer"  and  to  "provide  an  economic  impact  assessment  of  the  effects 
of  the  introduction  of  the  AMTESS  into  the  Army  maintenance  program" 
(p.  26).  The  RFP  prescribed  the  front-end  analysis  procedures  to  be 
conducted  for  the  conceptuai  design  of  the  training  device,  although 
no  reference  was  made  to  any  particular  method  or  model  (e.g.,  ISD) 
for  conducting  the  FEA. 

The  RFP  for  AMTESS  I  was  specific  in  terms  of  the  type  of 

training  device  desired.  The  device  was  to  be  computer-based  with 
flexible,  easy  to  operate,  easy  to  program  software.  The  device  was 
to  include  a  hands-on- train ing  capability.  The  device  .was  to  be 

modular  so  that  components  could  be  interchanged  easily. 

The  RFP  pointed  out  the  Army's  philosophy  for  the  use  of 

AMTESS.  It  stated  that  the  AMTESS  concept  "is  seen  as  an  adjunct  to 
primary  instruction"  and  as  such  "it  should  be  designed  to  be  a  cost 
and  training  effective  system  supplementing/modifying  the  maintenance 
training  presently  taught"  (p.  26). 

The  RFP  presented  three  specific  tasks  to  be  performed  "prior 
to"  the  design  of  the  training  device.  The  three  traditional  front- 
end  analysis  procedures  included  task  analysis,  training  requirements 
analysis,  and  fidelity  requirements  analysis.  A  -fourth  task,  design 
of  the  device  concept,  was  to  Include  an  initial  plan  for  evaluating 
the  device  and  an  economic  assessment  of  the  device  in  terms  of 

life-cycle  costs  over  a  15  year  period.  Each  of  the  four  tasks  is 
described  in  more  detail  below. 

(1)  Task  Analysis  The  purpose  of  the  task  analysis  was  to 
select  two  sets  of  tasks,  one  set  from  three  automotive  maintenance 
.-iOSs  (63B,  C,  and  H)  and  the  other  set  from  two  missile  maintenance 
MOSs  (24C  and  K).  These  tasks  would  be  the  tasks  to  be  tested  on  the 
prototype  device. 


26 


The  instructions  in  the  RFP  were  to  perform  two  separate  task 
commonality  analyses,  one  for  each  maintenance  specialty  (automotive 
and  missile).  Contractors  were  tasked  to  "classify  into  sets  those 
tasks  that  require  the  same,  or  very  similar  behaviors  with  the  same 
class  of  equipment”  (p.  28).  For  each  set  of  common  tasks, 
contractors  were  instructed  to  "select  sample  tasks  to  represent  the 
whole  set"  (p.  28).  From  these  sample  tasks,  contractors  wouM  select 
a  sufficient  number  to  be  taught  during  AMTESS  II  using  the  prototype 
training  device.  The  RFP  indicated  that  lists  of  tasks  could  be 
derived  from  government  furnished  data  such  as  Soldier's  Manual, 
Commander's  Manual,  applicable  maintenance  manuals,  and  Program  of 
I nstructlon. 

The  RFP  did  not  define  what  constituted  "commonality”  for 
purposes  of  classifying  tasks.  The  only  guidance  offered  for 
selecting  sample  tasks  from  the  commmon  classifications  was  to  select 
tasks  that  "from  a  cost  and  training  effective  viewpoint,  best  lend 
themselves  to  simulation"  (p.  28). 

(2)  Training  Requirements  Analysis.  The  purpose  of  this  task 
was  to  "structure  a  mini-program  of  instruction  (MPOl)  to  be  used  in 
the  test  and  evaluation"  (p.  28)  of  the  training  devices.  Essentially, 
the  MPOl  was  to  be  designed  to  train  the  maintenance  tasks  selected 
during  the  task  analysis.  The  MPOl,  according  to  the  RFP,  should 
interface  with  present  instruction.  However,  the  MPOl  was  not 
required  to  parallel  the  existing  POI.  The  primary  purpose  of  the 
MPOl  was  to  "support  the  test  of  the  hardware  concept." 

'  The  MPOl  was  to  describe  the  role  of  the  student,  the  role  of 
the  instructor,  and  the  role  of  the  simulator.  It  also  was  required 
to  include  how  the  time  of  instruction  would  be  spent  by  the  student. 

(3)  Fidelity  Requirements  Analysis.  The  purpose  of  this  task 
was  simply  to  determine  the  degree  and  kinds  of  simulation  fidelity 
required  by  the  training  device  to  train  the  tasks  selected  in  Task  1 
following  the  MPOl  outlined  in  Task  2. 


(4)  Design  of  Modular  Systems.  The  final  task  of  the  effort 
was  to  "develop  a  cost  and  training  effective  concept  for  a  modular 
maintenance  training  system  to  be  used  at  institutional  and  unit 
training"'  (p.  29).  Along  with  the  design  of  the  device,  the 

contractors  were  instructed  to  provide  the  following  three  additional 
pieces  of  information: 

•  Recommendations  on  how  the  prototype  model  of  the  concept  can 
be  evaluated 

•  Economic  assessment  of  the  device  over  15  years  and 

■  •  Trade-off  analysis  concerning  the  fidelity,  expectations  of 

transfer  of  training,  and  the  relative  costs  involved  with  the 
concept 

c.  AMTESS  I  Kick-off  Meeting.  This  meeting  was  held  over  the 
three  day  period  18-20  Sep  79.  A  major  agenda  item  entailed  each 
contractor  presenting  its  contract  performance  plan  to  the  Joint  Study 
Advisory  Group  members.  During  their  presentation,  contractors  were 
encouraged  to  raise  any  concerns  or  questions  about  the  AMTESS  I 
effort.  Contractors,  in  turn,  were  made  aware  of  the  procedures  for 
obtaining  relevant  documents  and  information  from  the  Army, 
particularly  from  the  schools.  Contractors  were  cautioned  about 
following  advice  from  government  representatives  except  for  authorized 
contracting  agents.  Indeed,  it  was  stressed  to  the  Contractors  that 
the  government  was  not  going  to  give  directions  and  that  the 
Contractors’  charge  was  to  make  decisions  based  on  cost  and  training 
effectiveness  as  ascertained  by  them  during  the  course  of  the 
contract. 

The  kick-off  meeting  also  discussed  contract  milestones  and 
prototype  test  issues.  Concerning  this  latter  item,  the  JSA6  agreed 
that  test  issues  should  be  identified  early  in  AMTESS  I  to  insure  an 
adequate  test  plan. 


d.  ARI  Eva-lua+ion  Plan.  ARI  agreed  to  be  Involved  in  the 
evaluation  of  the  AMTESS  training  devices  and  acknowledged  this 
agreement  by  signing  a  Memorandum  of  Understanding  with  PM  TRADE  on  7 
December  1979.  ARI  was  provided  with  a  general  overview  of  the 
prototype  devices  and  the  general  conditions  under  which  they  were  to 
be  evaluated  and  responded  on  22  January  1980  with  a  preliminary  test 
plan  for  evaluating  the  training  devices.  The  plan  entitled  "Training 
Effectiveness  Testing  for  AMTESS  I"  was  a  necessarily  general  plan 
which  proposed  a  test  design  encompassing  80  groups.  In  the  absence 
of  a  specifically  agreed  upon  set  of  test  objectives,  the  test  plan 
assumed  the  e\.jluation  would  include  two  (2)  simulator  models,  two  (2) 
locations  (unit  and  school),  and  instruction  on  four  (4)  pieces  of 
eauioment.  The  test  plan  proposed  to  test  the  following  five  groups 
of  subjects: 

1.  Control  g-oup  -  A  sample  of  subjects  who  had  received  no 
maintenance  training  at  all. 

2.  Baseline  group  -  A  sample  of  subjects  who  had  received 
current  conventional  maintenance  training. 

3.  Treatment  group  I  -  A  sample  of  subjects  who  had  received 
training  on  a  prototype  device. 

4.  Treatment  group  II  -  A  sample  of  subjects  who  had  received 
current  conventional  t'-aining  augmented  by  the  use  of  methodological 
innovations  incorporated  in  the  simulator  system  that  can  be  employed 
without  the  simulator  harware  or  software  iiself. 

5.  Expert  group  -  A  sample  of  "expert"  mechanics  used  to 
ascertain  the  feasibility  of  the  criterion  tasks  and  to  provide  an 
upper  edge  to  the  distribution  of  the  performance  scores. 

The  test  design  Included  broad  criteria  for  assigning  subjects 
to  test  conditions,  and  a  few  criterion  measures  of  performance. 
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PM  TRADE  recommended  modifications  to  the  ARI  test  plan  (6 
March  19R0)  in  the  following  areas: 

•  Reducing  the  number  of  treatment  conditions  from  80  to  a  more 
manageable  number  for  field  testing. 

•  Developing  a  specific  method  for  matching  treatment  group 
subjects. 

•  Developing  a  standardized  performance  based  maintenance  test  to 
be  used  as  a  performance  criterion  for  evaluating  the  tasks 
trained  in  AMTESS. 

PM  TRADE  attempted  to  provide  additional  detail  only  with 
regard  to  reducing  the  number  of  treatment  conditions.  The  ARI  test 
plan  assumed  that  two  simulators  would  be  evaluated  at  two  levels 
(unit  and  school)  with  each  simulator  being  used  to  train  four  pieces 
of  equipment.  Given  these  assumptions,  each  treatment  or  control 
group  tested  would  actually  result  in  16  treatment  conditions  or  cells 
in  the  design.  PM  TRADE  suggested  that  the  "Control  group,"  Treatment 
group  II,"  and  the  "Expert  group"  were  unjustified  and  could  be 
eliminated.  That  would  leave  two  test  groups  ror  a  total  of  32  test 
cond i tions . 

Specific  methods  for  matching  treatment  group  subjects  or 
developing  performance  criteria  were  not  advanced  by  PM  TRADE  at  this 
time. 


e.  Grumman  AMTESS  I  Reports.  Grumman  was  one  of  the  two 
contractors  from  AMTESS  I  who  was  selected  to  build  a  prototype  model 
of  their  device  concept.  Grumman  conducted  the  three  FEA's  prescribed 
in  the  AMTESS  I  RFP  along  with  a  cost  effectiveness  analysis  and  a 
brief  recommendation  for  evaluating  their  device.  The  three  primary 
FEA  procedures,  task  analysis,  training  requirements  analysis,  and 
fidelity  analysis  are  described  below,  along  with  a  summary  of  the 
device  concept  Grumman  proposed. 
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(1)  Task  Analysis.  Grumman  conducted  a  comprehensive  task 
analysis.  The  task  analysis  consisted  of  the  following  six 
interrelated  steps: 

•  Identify  population  of  tasks 

•  Classify  tasks  based  on  behavior  performed 

•  Assess  current  Army  training  needs 

•  Identify  candidate  tasks 

•  Identify  equipment  commonalities 

•  Select  tasks 

Grumman  identified  candidate  tasks  by  reviewing  pertinent 
government  documents.  For  the  automotive  MOSs,  they  classified  tasks 
on  the  basis  of  five  functional  classifications  including  Inspect, 
Troubleshoot,  Test  and  Adjust,  Remove/Replace,  and  Vehicle  Operation. 
They  identified  171  common  tasks  for  the  three  automotive  MOSs. 

Through  discussion  with  training  school  personnel  and  observation  of 
trainees,  Grumman  identified  79  tasks  of  the  171  common  tasks  which 
they  defined  as  having  the  highest  priority.  From  these  79  tasks  they 
selected  '^1  which  best  lent  themselves  to  simulation.  These  31  tasks 
were  selected  for  testing  during  AMTESS  II  on  the  Grumman  breadboard 
s imu I ator . 

For  the  missile  MOSs,  a  total  of  50  tasks  were  identified. 
These  30  tasks  could  be  divided  generally  into  performance  of 

operational  checks,  and  troubleshooting  and  repair  of  transmitter 
circuits.  The  repair  tasks  were  further  subdivided  by  levels  of 

difficulty  and  the  five  basic  circuits  of  the  transmitter. 

A  major  finding  of  the  missile  maintenance  task  analysis  was 
that  the  tasks  examined  did  not  consist  of  easily  d i scr i mi  nab  I e 

discrete  tasks,  but  rather  of  a  prescribed  logical  sequence  of  tasks. 
Therefore,  Grumman  selected  five  sequences  of  tasks  for  inclusion  in 
the  simulator.  These  task  sequences  were  "selected  carefully  in  order 
to  Insure  that  each  of  them  could  be  generalized  to  other  areas  of  the 
transmitter"  (Campbell,  Stonge,  Cothran,  Anaya,  and  Sicuranza,  19R0). 


It  is  important  to  note  that  in  selecting  the  tasks,  Grumman 
adopted  the  philosophy  that  ’’the  initial  implementation  of  AMTESS  is 
intended  to  be  supplementary  to  existing  training"  (Campbell,  Stonge, 
Cothran,  Anaya,  and  Sicuranza,  1980).  As  a  result,  many  of  the  tasks 
selected  were  tasks  with  a  high  training  priority  which  were  not 
currently  being  trained  at  the  schools.  Table  2  contains  a  list  of 
tasks  selected  by  each  contrac^-or .  ' 

(2)  Training  Requirements  Analysis.  Grumman's  training 
requirements  analysis  began  with  a  needs  analysis  to  identify  major 
problem  areas  involved  with  existing  training.  Those  problems  for 
which  AMTESS  might  provide  a  solution  were  identified  and  discussed, 
following  the  needs  analysis,  separate  curricula  analyses,  one  for 
missile,  and  one  for  automotive,  were  conducted.  The  analyses 
compared  and  contrasted  the  current  training  procedures  for  each  MOS 
including  curriculum  content,  lesson  organization,  and  teaching 
methodology. 

Based  on  the  training  requirements  analysis,  Grumman  developed 
an  MPOl  for  implementation  during  AMTESS  II.  They  selected  a  total  of 
9  automotive  tasks  (from  the  initial  set  of  31)  and  6  missile  tasks 
(from  the  initial  set  of  30)  for  training  in  the  MPOl.  Grumman's 
discussion  of  the  MPOl  included  an  outline  of  the  lesson  specification 
training  the  selected  tasks.  The  lesson  included  the  following 
i  nformation; 

•  Objectives 

•  Tasks  to  be  trained 

•  Relationship  to  currently  trained  curriculum 

•  Prerequisite  skills  and  knowledges 

•  Allocation  of  media 

•  Management  and  admi nstrative  requirements 

•  Learn i ng  abilities 

•  Selected  training  strategies 

•  Estimated  time  for  completion 
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TABLE  2 


AUTOMOTIVE  TASKS  SELECTED  FOR  SIMULATION  BY  CONTRACTORS 

Troubleshoot 


Engine 

X 

X 

x(M60) 

Cooling  System 

X 

Oil  Filter 

X 

Oil  Pump 

Suspension  (Track) 

X 

X 

Automatic  Transmission 

X 

Suspension  (Wheel) 

X 

Remove  &  Replace 

Thermostat 

Water  Pump 

Alternator 

Starter  Motor 

Fue 1  Pump 

Tank  Engine  (M60) 

Steering  Hydraulic  Hoses 


Water  Pump  Belt  x 

Alternator  Belt  x 

Alternator  Output  x 

Valves  (M60) 

Transmission  Bands  (M60) 

Power  Steering  Cylinder  (M809) 

Perform 

DC  Voltage  Test  (MM) 

Continuity  Test  (MM) 

DC  Voltage  Test  (LUCT) 

DC  Current  Test  (LUCT) 

Resistance  (MM) 

AL  Voltage  Test  (MM) 

Inspect 


Engine  x 
Brakes  x 
Track  Vehicle  Suspension  System  x 
Gas  Filter  System  x 


XX  X 

XX  X 

X  X 

X  X 

X  X 

X  X 


X  X 

X 

X 


Seville  Honeywell  Grumman  Hughes 


TABLE  2  (Cont.) 


MISSILE  TASKS  SELECTED  FOR  SIMULATION  BY  CONTRACTORS 


Troubleshoot 


Seville  Honeywell  Grumman  Hughes 


Transmitter  RF  Generation 
Transmitter  Noise  Generation 
Transmitter  Modulation  Filament 
Transmitter  Group  Low  Voltage 
High  Voltage  Power  Supply 
Transmitter  Test  Set 
Low  Voltage  Power  Supply 
Degeneration  Rack 
Transmitter 

Remove /Install 

Power  Amplifier  Filament 
Power  Supply 

Master  Oscillator  Filament 
Power  Supply 
High  Voltage  Regulator 
Amplifier  Modulator  Oscillator 
Power  Amplifier  Tube 
Master  Oscillator 

Adjust 

Master  Oscillator  Frequency 
Transmitter 


Power  Amplifier  Filament 
Power  Supply 
High  Voltage  Regulator 
Master  Oscillator 

Repair 

Range  &  Coding  Amplifier  Oscillator 
Low  Voltage  Power  Supply 
Transmitter  Circuits 
Transmitter  Function  Circuits  ' 
Bridge  Milling  Amplifier 
and  Cavity  Tuner 

Maintain 

Noise  Cancellation  Circuits 
Modulation  Circuits 

Perform 


Weekly  Operation  Checks 
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While  Grumman  provided  outlines  of  lesson  specifications,  it 
left  to  AMTESS  II  the  actual  development  of  detailed  lesson  plans. 


(3)  Fidelity  Requirements  Analysis.  Grumman  considered  five 
major  factors  in  the  fidelity  analysis  including: 

•  Degree  of  dimensionality, 

•  The  types  of  cues  the  AMTESS  trainer  must  provide  the  student, 
and  the  inputs  it  must  receive  from  the  student 

•  Relative  importance  of  those  cues  and  inputs  to  the  trainee's 
ability  to  learn  the  task 

•  The  fidelity  required  in  cue  presentation  to  permit  trainees  to 
make  necessary  discriminations  and 

•  Dynamic  response  fidelity  required 

Each  of  the  tasks  selected  in  the  task  analysis  was  examined 
separately  with  respect  to  these  criteria.  The  results  indicated  that 
virtually  all  tasks  could  be  simulated  using  a  2-D  media  presentation. 

(4 )  Device  Concept. 

i.  Hardware  Des i gn .  Grumman  identified  three 
alternative  configurations  for  the  simulator  dev'Ice.  One  design 
included  a  set  of  student  substations  feeding  off  a  central  instructor 
station  consisting  of  a  computer,  mass  storage,  terminal,  and  printer. 
Student  substations  would  contain  no  Independent  computing  power,  so 
all  processing  would  take  place  in  the  central  processing  unit  of  the 
instructor  station.  A  second  system  design  was  similar  to  the  first 
except  that  the  student  substations  would  include  computing  power.  The 
central  processing  unit  would  be  used  primarily  for  information 
storage . 


Grumman  determined  that  flaws  existed  with  each  of  these  system 
concepts.  Hence,  a  third  configuration  was  devised,  a  stand-alone 
unit.  The  stand-alone  unit  included  a  2-D  workstation  consisting  of  a 
computer,  student  input  and  display  subsystem,  an  instructor  CRT  and 
printer,  a  mass  storage  unit  and  a  T-D  simulation  subsystem.  The  2-D 
trainer  had  the  following  components: 

•  Microcomputer 

•  CRT 

•  Videodisc  &  TV  monitor  with  speakers 

•  Touch  panel  and  keyboard 

•  Floppy  disc 

•  Hard  disc 

•  Pr i nter 

•  Standard  RS232  interfaces 

•  Parallel  data  address  bus  to  interface  with  T-D  simulator 

Grumman  planned  two  3-D  simulator  modules,  one  for  the 
automotive  MOSs  and  one  for  the  missile  MOSs.  The  automotive 
simulator  was  designed  to  allow  checking,  diagnosing  and 
troubleshooting  the  engine,  electrical  system,  starter,  battery, 
generator,  regulator,  and  representative  wiring  and  gauges  for  an 
automotive  system.  The  missile  simulator  was  designed  to  permit 
transmitter  checks  and  hands-on  diagnosing  and  troubleshooting  for  the 
Improved  High  Power  Illuminator  (IHIPIR).  Table  3  contains  a  summary 
of  the  key  device  features  proposed  by  each  of  the  four  contractors. 

ii.  Evaluation  Plan.  Grumman 's  plan  for  evaluating 
their  device  involved  the  basic  elements  of  an  experimental  design. 
Students  taking  part  in  the  evaluation  would  be  matched  on 
prerequisites  to  be  determined  at  a  future  date.  Students  would  be 
randomly  assigned  to  one  of  three  groups:  a  control  group,  treatment 
group  1,  and  treatment  group  2.  Treatment  group  1  would  receive 
existing  training,  and  treatment  group  2  would  receive  AMTESS 
training.  The  control  group  would  not  receive  existing  or  AMTESS 
training  (note  that  Grumman' s  approach  was  to  be  supplemental  of 
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existing  training.)  Following  training,  all  students  would  be 
required  to  perform  tasks  (to  be  determined)  on  actual  equipment. 
Criterion  measures  for  evaluating  this  performance  would  be  developed, 
and  the  results  would  provide  evidence  of  transfer-of-tra i n i ng  • 

iii.  Economic  Assessment.  Grumman  used  a  threefold 
approach  to  estimate  device  costs  over  a  fifteen  year  life  cycle. 
First,  based  upon  their  preliminary  system  engineering  design,  Grumman 
identified  a  number  of  potential  "off-the-shelf"  components  for  use. 
Next,  these  alternative  components  were  evaluated  on  the  basis  of 
anticipated  performance  and  estimated  costs  associated  with  that 
performance,  including  design,  production,  and  maintenance  costs. 

In  evaluating  the  alternative  device  components,  anticipated 
performance  was  measured  via  a  rating  scheme  developed  by  Grumman 
which  included  a  user  evaluation  component.  The  relative  costs  of 
each  component  were  measured  according  to  a  life  cycle  cost  model 
which  Grumman  termed  the  AMTESS  Cost  Model  System  (ACMS).  This  model 
is  a  "linked  and  graded"  model  first  developed  for  the  U.S.  Navy.  A 
"graded"  model  is  one  which  becomes  increasingly  complex  to  reflect 
the  quality  and  quantity  of  data  available.  The  term  "linked"  refers 
to  the  fact  that  the  models  may  be  related  to  one  another  at  different 
grades . 


The  third  step  in  the  economic  assessment  involved  solutions  of 
the  ACMS  for  the  specific  device  components  selected.  The  model 
included  estimates  of  Research  and  Development  Costs,  Investment 
Costs,  Operating  and  Support  Costs,  Consumption  Costs  (e.g.,  fuel). 
Depot  Maintenance,  Other  Direct  Support  Operations,  and  Indirect 
Support  Operations.  The  economic  assessments  were  made  under  the 
assumption  of  producing  300  2-D  workstations,  and  20  each  of  15 
different  kinds  of  3-D  modules.  Life  cycle  is  defined  as  15  years. 
The  average  life  cycle  cost  per  system  including  software  was 
estimated  at  $93,943  in  1978  dollars. 


f.  Honeywell  AMTESS  I  Reports.  Honeywell  participated  in 
AMTESS  I  but  was  not  selected  to  develop  a  prototype  device  based  on 
their  AMTESS  I  concept.  In  response  to  the  AMTESS  I  RFP,  Honeywell 
performed  the  task  analysis,  training  requirements  analysis,  fidelity 
analysis,  and  they  developed  a  device  concept  complete  with  an 
evaluation  plan.  These  are  detailed  below. 

(1)  Task  Analysis.  Honeywell  enumerated  all  the  tasks 
involved  in  the  five  MOSs  and  classified  them  into  the  following 
categories:  (See  Behm,  Johnson,  Graf,  Hirshfeld,  &  McAleese,  1980a&c) 

Automot i ve  Missile 

Troubleshooting  Fault  Isolate  and  Repair 

Align  and  Adjust  Align  and  Adjust 

Remove  and  Replace  Inspect  and  Service 

I nspection 

In  conducting  the  commonality  analysis,  Honeywell  used  the 
General i zab I e  Job  Proficiency  Matrix  (GJPM),  a  technique  they  had 
developed  previously.  The  GJPM  identifies  major  areas  of 
commonalities  across  tasks  and  provides  a  systematic  and  efficient 
basis  for  selecting  tasks  and  sub-tasks  appropriate  for  training. 

Following  a  training  analysis  to  identify  Important  training 
tasks  and  a  cost  analysis  to  determine  cost  effective  training  tasks, 
Honeywell  developed  a  simulator  va'ue  index  for  the  tasks.  Using  the 
index,  Honeywell  selected  the  specific  automotive  and  missile  tasks 
for  their  MPOI.  (See  Table  2). 

(2)  Training  Requirements  Analysis.  Honeywell  developed  26 
individual  lesson  nradules  for  the  automotive  tasks  and  46  for  the 
missile  tasks.  Each  module  in  the  MPOI  was  presented,  at  a  minimum, 
with  -a  lesson  title,  performance  criterion  for  the  module,  enabling 
objectives  (i.e.,  maintenance  requirements  necessary  to  support  the 
performance  criterion),  and  a  time  estimate  for  completing  the  module. 
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Some  of  the  modules  were  developed  to  supplement  exist! np 
training  while  others  were  designed  to  replace  it.  Honeywell 
presented  a  detailed  matrix  showing  the  relationship  between  their 
MPn I  and  the  current  POI.  Honeywell  also  provided  a  discussion  of  the 
roles  of  the  student,  instructor,  and  the  simulator  in  AMTESS. 

fS)  Fidelity  Reguirements  Analysis.  The  heart  of  Honeywell's 
fidelity  requirements  analysis  was  a  fidelity  algorithm  developed  for 
a  previous  contract  (Plocher  and  Koch,  1979).  Generally,  the 
algorithm  entailed  analyzing  tasks  with  regard  to  the  following 
decision  points: 

•  What  must  be  learned: 

-  procedural  sequence? 

-  motor  skill? 

-  both? 

•  Do  man ipu I ations  involve: 

-  simple  motor  skills? 

-  complex  perceptual -motor  skills? 

•  Is  the  stimulus  display: 

-  discrete? 

-  dynamic? 

The  analysis  also  took  into  account  information  on  requirements 
in  the  areas  of  olfactory,  auditory,  visual,  tactile,  and  kinesthetic 
cues  for  purposes  of  developing  simulation  realism. 

Honeywell  divided  fidelity  into  seven  categories  ranging  from 
simple  2-n  fidelity  using  a  video  CRT  driven  by  a  computer  to  high 
fidelity  T-D  simulators.  The  fidelity  analysis  entailed  examining 
each  task  selected  for  the  MPOl  with  regard  to  the  decision  points  and 
classifying  each  task  into  one  of  the  seven  fidelity  categories. 


The  results  of  the  fidelity  requirements  analysis  indicated 
that  more  than  85  percent  of  all  tasks  could  be  trained  using  2-D 
media  with  10  percent  requiring  a  low  level  fidelity.  Only  5 

precent  of  the  automotive  tasks  and  1  percent  of  the  missile  tasks 
were  found  to  require  hiqh  fidelity  simulators. 

( 4 )  Device  Concept. 

i.  Hardware  Design.  The  Honeywell  device  included  a 
computer  system  (Primary  AMTESS)  with  "plug-in"  !^-D  modules  which 
attach  to  the  Primary  AMTESS  (Behm,  Johnson,  Graf,  Hirshfeld, 
McAleese,  1980b).  The  Primary  AMTESS  had  the  following  components: 

( See  also  Table  ) 

•  Microcomputer  -  (DEC  LSI -11) 

•  Videodisc 

•  Color  CRT 

•  Touch  screen/digit i:ier 

•  Black  &  white  CRT  with  keyboard 

•  Dual  cartridge  tape  drive 

•  Hard  disc  drive  (5.2  megabytes) 

•  Dot  matrix  printer 

•  Interfaces 

Honeywell  proposed  four  T-D  simulators  including  the  foliowing: 

•  Radar  Servo  Adjustment 

•  Automobiie  Valve  Adjustment 

•  Power  Steering  Cylinder 

•  Tank  Transmission  Servo 

ii.  Evaluation  Plan.  Honeywei i  sketched  out  a  brief 
evaluation  plan  which  involved  comparing  AMTESS  trained  perfor'^ance 
with  the  current  Army  training.  Honeywell  esvimated  that  about  twelve 
weeks  would  be  required  at  an  installation  to  train  with  and  evaluate 
each  device  using  two  devices  at  an  installation. 
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iii.  Economic  Assessment.  Honeywell's  approach  to  the 
economic  assessment  was  similar  to  Grumman' s  in  a  number  of  respects. 
First,  based  on  the  preliminary  system  engineering  design,  Honeywell 
developed  a  list  of  possible  design  components.  These  possible 
components  were  subjected  to  a  trade-off  analysis  which  addressed  the 
component's  reliability,  estimated  cost,  and  the  module's  capability 
to  satisfy  the  greatest  number  of  requirements  of  the  system. 

Following  the  selection  of  the  superior  components  of  the 
system  via  their  trade-off  analysis,  Honeywell  identified  an 
appropriate  military  life  cycle  cost  model  for  application  at  the 
current  stage  of  the  AMTESS  Program.  Honeywell  selected  a  model 
developed  by  the  USAF  ASD/AFALD  LCC/OTC  at ' Wr ight-Patterson  Air  Force 
Base,  Ohio. 

In  general  the  model  estimates  the  development,  procurement, 
and  ownership  costs  of  the  system.  Honeywell's  estimates  assumed  the 
production  of  300  systems  and  estimated  life  cycle  costs  over  15 
years.  Honeywell  estimated  an  average  system  cost  of  $87,562  in  1978 
dollars.  These  costs  could  be  subdivided  into  approximately  $3,084 
for  development,  $51,739  for  procurement,  and  $32,739  for  ownership. 

g.  Hughes  Aircraft  Company  AMTESS  I  Reports.  Hughes 
participated  in  the  AMTESS  I  effort,  but  was  not  selected  to  develop  a 
prototype  dgvice  in  AMTESS  II.  To  arrive  at  their  device  concept  in 
AMTESS  I,  Hughes  performed  the  three  prescribed  FEA's  and  provided  an 
evaluation  plan  for  the  devices.  These  activities  are  described  in 
detail  in  the  following  paragraphs. 

(1)  Task  Analysis.  Hughes  reviewed  the  list  of  tasks 
performed  for  each  of  the  five  MOSs .  Hughes  identified  one  general 
factor  encompassing  all  tasks,  a  Test/Operate  factor;  and  three 
"active  clusters"  defined  as  "Inspect/Troubleshoot", 
"Remove/Rep  I  ace/I nstal I /Service",  and  "Ad just/A I i ne(sic)"  (Dickson, 
Phillips,  Queen,  and  Toth,  1980  (a). 


"^ased  on  the  clusterinu  activities,  four  tasks  were  selected 
reoresent'nq  "the  total  universe  of  tasks  adequately"  (Dickson, 
Phi  I  I  ins,  Oueen  and  Toth,  IQftO  (a).  These  four  tasks  were; 

•  Trouble  Shoot  Vehicle  Enqine  -  Automotive 

•  Remove  Power  Plant  -  Automotive 

•  Adjus+  Transmission  Band  Services  -  Automotive 

•  Trouble  Shoot  and  Adjust  Transmitter  -  Missile 

Three  tasks  were  automotive  tasks  selected  to  represent  all 
tasks  performed  in  the  three  automotive  MOSs.  One  task  was  a  missile 
task  selected  to  represent  all  missile  tasks  from  the  two  missile 
MOSs.  (See  Table  2)  . 

(?)  Training  Requirements  Analysis.  Hughes  training 
requirements  analysis  encompassed  essentially  four  components.  These 
included  a  "ml crodecomposition"  of  the  four  tasks  into  their  basic 
elements,  determination  of  training  media,  outline  of  functional 
training  requirements,  and  discussion  of  the  role  of  the  instructor. 

"Microdecomposition"  entailed  splitting  tasks  into  basic 
learning  elements  and  simulation  elements.  Basic  learning  elements 
were  defined  as  the  enabling  skills  and  processes  a  trainee  needs  in 
order  to  perform  a  particular  maintenance  task.  Simulation  elements 
were  defined  as  those  sensory  cues  required  to  perform  the  maintenance 
activity.  Sensory  cues  include  initiating  stimuli,  overt  responses  of 
the  trainer,  and  equipment  responses  to  the  trainees'  inputs. 

Hughes  "microdecomposed"  the  four  tasks  selected  in  the  task 
analysis  into  TTfi  individual  elements.  Hughes  further  analyzed  these 
elements  and  found  that  most  (72.5  percent)  required  cognitive  skills, 
while  only  27.5  percent  required  a  motor  skill. 

Based  on  this  analysis,  Hughes  decided  that  the  primary 
training  medium  could  be  a  generic  2-D  trainer.  This  2-D  trainer 
would  be  a  computer-based  audio-visual  system.  This  device  concept  is 
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described  in  mone  de+all  later. 

Hughes  then  listed  severs!  functional  requirements  for  the 
training  program.  Among  these  requirements  were  the  following: 

•  Training  should  be  learned  and  applied  immediately  rather  than 

learned  and  not  applied  for  a  long  p<--.“iod  of  time. 

•  Training  should  be  sequenced  from  simple  to  complex  tasks. 

•  Training  should  be  self-paced  based  on  trainee's  performance. 

Basic  instruction  would  include  four  modes: 

•  Preview  the  task 

•  Practice  the  task 

•  Testing  of  trainee  on  ’task 

•  Periodic  review  of  task 

Hughes'  training  requirements  analysis  concluded  with  several 
scenarios  for  the  instructor.  It  is  important  to  note  that  Hughes  did 
not  present  an  MPO I  in  their  report.  Instead,  they  delayed  a 
comprehensive  curriculum  review  and  MPOl  development  until  AMTESS  II. 

(3)  Fidelity  Requirements  Analysis.  Hughes'  fidelity  analysis 
considered  both  physical  and  psychological  fidelity.  Psychological 
fidelity  was  defined  as  how  fully  the  training  environment  exercises 
mental  skills  required  in  performing  job  tasks.  Hughes'  philosophy 
was  to  reduce  or  eliminate  non-essential  elements  of  physical  fidelity 
while  retaining  high  psychological  fidelity. 

Hughes'  approach  to  the  fidelity  analysis  was  to  examine  each 
of  338  individual  elements  and  rate  the  fi'del  ity  needed  along  nine 
general  and  30  specific  fi.del  ity  dimensions.  The  results  of  the 
fidelity  analysis  indicated  that  most  fidelity  requirements  could  be 
achieved  using  a  generic  2-D  trainer.  Six  subtasks  were  identified 
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which  would  require  an  additional  piece  of  equipment  with  a  hiqher 
deqree  of  fidelity  beyond  the  qeneric  2-D  trainer. 

The  fidelity  analysis  confirmed  the  findings  of  the  task 
analysis  and  training  requirements  analysis  that  most  tasks  could  be 
trained  on  a  2-0  computer-based  trainer. 

( 4 )  Device  Concept. 

i.  Hardware  Des i gn .  Hughes  envisioned  that  their 
AMTESS  training  system  would  encompass  a  family  of  four  components. 
The  workhorse  of  the  family  would  be  a  qeneric  2-D  trainer  which  would 
be  used  for  87  percent  of  training.  It  would  be  supplemented  by  three 
additional  componen+s,  a  Generic  Tactile  Trainer,  Hardware  Specific 
Tactile  Trainers,  and  Actual  Equipment  (Dickson,  Phillips,  Queen, 
Toth,  &  Desantis,  1P80). 

In  selecting  equipment  for  the  AMTESS  device,  Hughes’  primary 
concern  was  that  the  majority  of  the  hardware  components  were  to  be 
military  or  commercial  off-the-shelf  items  with  proven  reliability, 
availability,  size,  weight,  and  compatibility  with  approved  military 
hardware.  Hughes  also  tended  to  prefer  equipment  with  which  they  had 
previous  working  experience. 

Based  on  their  analysis,  Hughes  designed  the  2-D  trainer  tc 
include  the  following  components.  (See  also  Table  T) , 

•  DEC  LSI-11/2T  microcomputer 

•  Winchester  hard  disc  for  permanent  mass  storage 

•  Streaming  tape  for  removab I e,  mass  storage 

•  Serial  and  parallel  input/output  interfaces 

•  Floppy  discs  for  removable  mass  storage 

•  Color  graphics  display 

•  Random  access  video  recorder  for  removable  video/audio  storage 

•  Digital  audio  synthesizer 

•  Interactive  panel 
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•  °r inter 

•  Instructor's  terminal  such  as  a  Diablo  or  i  nter / keyboard 
term i na I 

•  General  purpose  interface  bus  to  interface  with  the  hardware 
specific  tactile  trainer 

•  Traininq  software 

Generic  Tactile  Trainer  -  The  purpose  of  this  component  was  to 
assist  in  training  motor  skills  which  require  tactile  training.  This 
would  include  tasks  such  as  the  use  of  selected  tools  (e.g.,  wrenches, 
calipers)  and  selected  general  actions  such  as  soldering.  The  actual 
configuration  of  the  Generic  Tactile  Trainer  was  left  for  the  future 
until  "further  classification  of  the  tasks  to  be  trained  is  made" 
(Dickson,  Phillips,  Queen,  and  Toth,  IPROa).  While  specification  of 
the  actual  configuration  was  postponed,  Hughes  did  note  that  the 
Generic  Tactile  Trainer  would  not  be  computer-based. 

Hardware  Specific  Tactile  Trainers  -  These  items  constituted 
special  pieces  of  hardware  to  be  used  in  only  a  few,  limited 
;ircumstances.  They  would  be  developed  only  for  those  few  motor  skill 
tasks  which  meet  several  conditions.  Hughes  did  not  provide 
specifications  for  this  component  because  of  the  uncertainty  of  the 
tasks  to  be  trained.  The  Hardware  Specific  Tactile  Trainers  were  not 
expected  to  be  computer-based. 

ii.  Evaluation  Plan.  Hughes'  evaluation  plan  consisted 
of  descriptions  of  various  scenarios  supplemented  by  evaluation 
recommendations.  Hughes  clearly  anticipated  a  number  of  difficulties 
in  evaluating  the  AMTESS  devices.  These  difficulties  ranged  from 
availability  of  subjects,  to  the  comparison  of  devices  between 
contractors,  to  the  specification  of  evaluation  objectives. 

Hughes  noted  that  the  major  problem  in  evaluating  AMTESS  was 
troop  support  considerations  for  "fielding  sufficient  training  hours 
on  AMTESS."  Hughes  pointed  out  that  the  existing  PO I  for  one 
automotive  MOS,  6TH,  included  T60  hours  of  instructional  time.  They 


argued  that  even  10  hours  of  AMTESS  instruction  would  only  amount  to  ? 
percent  of  the  existing  plan  of  instruction  (POl). 

''  second  problem  Hughes  foresaw,  was  how  to  make  a  fair 
comparison  between  AMTESS  and  conventional  training.  Hughes  proposed 
two  possible  designs.  One  design  would  involve  us i ng.  students  who  had 
completed  the  initial  part  of  the  existing  POl.  These  students  would 
be  randomly  assigned  either  to  continue  with  the  current  POl  or  to 
take  a  parallel  POl  offered  on  AMTESS.  Subsequent  to  completing  the 
POl,  students  in  both  groups  would  be  compared  to  determine  which 
training  situation  (AMTESS  or  conventional)  produced  the  superior 
student . 

A  second  design  would  involve  development  of  an  MPO I  unrelated 
to  the  existing  POl.  Trainees  would  learn  all  the  relevant  material 
using  either  AMTESS  or  i n  a  "control  condition"  using  training  methods 
similar  to  the  existing  POl. 

Hughes  indicated  that  they  were  unclear  as  to  whether  the 
government  evaluation  effort  would  compare  devices  between 
contractors.  They  noted  that  if  such  comparisons  were  to  be  made,  the 
devices  should  provide  training  on  at  least  one  common  task. 

iii.  Economic  Assessment.  Hughes  used  the  RCA  Price  84 
Program  for  estimating  life  cycle  costs.  Their  estimates  were 
confined  to  the  2-D  trainers  since  the  engineering  design  for  the  5-D 
trainer  had  not  been  completed  at  that  time.  The  RCA  Price  84  Program 
uses  inputs  for  engineering  and  manufacturing  costs  both  during 
development  and  production  of  the  2-D  system.  The  model  also  takes 
into  account  predictive  estimates  of  the  system’s  performance  such  as 
mean  time  between  failures,  support  removable  units,  and  support 
effort  requirements. 

Hughes  estimated  that  the  life  cycle  cost  of  the  2-D  trainer 
would  be  approximately  $75,000  in  1978  dollars.  While  Hughes  did  not 
estimate  the  costs  associated  withe  the  3-D  trainer,  they  did  indicate 
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that  these  costs  would  be  relatively  small  when  comoared  to  the  2-0 
tra I ner . 

h.  Sev  i  I  I e-Burtek  AMTESS  I  Reports.  Sev I  I  I e-Burtek ,  a 
consortium  consisting  of  the  Seville  Research  Corporation  along  with 
Burtek,  Inc.,  was  one  of  two  contractors  selected  from  the  four  AMTESS 
I  competitors  to  develop  a  working  device  in  AMTESS  II.  Seville  was 
the  primary  contractor  in  these  activities,  while  Burtek  provided 
engineering  and  design  support.  During  AMTESS  I,  Sev  i  I  I e-Burtek 
performed  the  three  prescribed  FEAs  (task  analysis,  training 
reguirements  analysis,  and  fidelity  analysis).  They  also  provided  a 
preliminary  design  of  the  device  along  with  cost  effectiveness 
documentation.  Sev i I  I e-Burtek  did  not  provide  a  systematic  evaluation 
plan  in  thei r  ,  report,  but  they  did  provide  a  brief  recommendation 
concerning  evaluation.  A  summary  of  the  front-end  analysis  activities 
is  provided  below.  A  complete  description  of  the  design  and 
development  of  the  Sev i I  I e-Burtek  device  is  presented  in  their  final 
reports  for  AMTESS  II  (Garlichs,  Miller  and  Davis,  19BTa,b,c). 

(1)  Task  Analysis.  Sev i I  I e-Burtek ' s  approach  to  the  task 
analysis  was  to  group  tasks  into  categories  such  that  the  training 
designed  for  any  one  task  in  the  group  would  apply 'Vo  all  tasks  in  the 
group.  Sev i I  I e-Burtek  identified  the  following  categories  into  which 
tasks  from  both  automotive  and  missile  MOSs  could  be  grouped. 

••  Inspect 

•  Troubleshoot 

•  Remove 

•  Rep  I  ace/ 1 nstal I 

•  Ad just/A I i gn 

•  Service 

•  Test 

It  should  be  stressed  that  while  these  categories  applied  to 
both  automotive  and  missile  MOSs,  the  tasks  from  each  area  were 
classified  separately  into  these  categories.  Once  the  tasks  were 
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orouDed  into  common  cateqories,  Sevi I le-Buntek  selec+ed  candidate 
■♦■asks  for  i  ncorpora'*' i  on  into  the  MPOl  .  In  selecting  tasks, 
Rev i I  I e-Burtek  chose  a  set  of  tasks  which  could  be  trained  using  a 
single,  simulator-  module.  As  noted  in  their  report,  Sev i I  I e-Burtek 
intended  the  implementation  and  evaluation  of  their  device  to  be 
stra i qhtforward : 

Criteria  for  task  selection  were  the  amenability  of  tasks  to 
development  of  a  unified  block  of  instruction  to  facilit'te 
evaluation,  the  minimization  of  resources  required  for  course 
development,  and  the  logistics  involved  in  implementation 
(Long,  Miller,  &  Garlichs,  19R0a). 

Sev  i  I  I  e-9urtek  selected  2"^  automotive  tasks  and  22  missile 
tasks  as  suitable  for  performance  with  AMTESS.  (See  Table  2). 

(2)  Training  Requirements  Analysis.  Sev i I  I e-Burtek  began  their 
analysis  by  reviewing  current  training  requirements,  procedures,  and 
operations.  This  review  included  examination  of  school  documen'^s, 
discussions  with  training  personnel,  and  observation  of  training. 

Unlike  Grumman,  who  believed  that  the  simulator  initially  would 
fill  a  supplementary  training  role,  Sev i I  I e-Burtek  believed  that  the 
simulator  would  assume  the  identical  role  of  the  actual  equipment 
trainer  (AET)  it  simulates.  That  is,  it  would  be  a  substitute  for  the 
AET.  Based  on  this  philosophy,  and  on  the  goal  of  being  able  to 
implement  the  simulator  in  a  straightforward  manner,  Sev i I  I e-Burtek 
concluded  that  "the  most  desirable  situation  would  be  a  course 
organization  and  structure  that  would  allow  one-to-one  substitution  of 
simulator  training  tasks  for  existing  tasks"  (Long,  Miller,  and 
Garlichs,  1980a).  Therefore,  Rev  i  I  I e-Burtek  developed  two  courses  (one 
for  the  missile  speciality,  and  one  for  the  automotive  speciality), 

each  of  which  could  be  substituted  directly  for  specific  portions  of 
existing  training. 


The  automotive  MPOl  was  desipned  for  the  Cummins  diesel  engine 
and  taken  specifically  from  MOS  63H10.  The  MPO I  was  presented  in 
considerable  detail  and  included  objectives  for  the  various  lessons, 
student  prerequisites,  lesson  plans,  and  actual  practice  exercises. 

The  missile  MPOl  was  desipned  generally  for  the  Improved  Hawk 
Firing  Section  Equipment  Maintenance  Course  and  specifically 
maintenance  on  the  IHIPIR.  The  MPOl  presentation  included  the 
objectives  of  each  lesson  along  with  practical  exercises  and  tests. 

(3)  Fidelity  Requirements  Analysis.  Rather  than  confine  their 
fidelity  analysis  to  the  tasks  selected  for  training  (as  indicated  by 
the  RFP),  Sev 1 1  I e-Burtek  decided  to  conduct  their  analysis  on  all 
maintenance  tasks.  In  the  analysis,  they  considered  the  fidelity  of 
tools  and  training  support  materials.  Fidelity  was  examined  along  two 
primary  dimensions,  physical  characteristics  (e.g.,  size),  and 
operational  characteristics  (e.g.,  audible  outputs). 

The  principal  cone  I usion '  of  the  fidelity  analysis  was  that  the 
simulator  hardware  must  look  like  the  equipment  it  simulates, 
disassemble  like  the  actual  equipment,  but  the  simulator  did  not  have 
to  operate  like  the  equipment. 

( 4 )  Device  Concept. 

i.  Hardware  Design.  The  Sev i I  I e-Burtek  device  concept 
featured  a  2-D  computer  based  simulator  coupled  with  a  3-D  system 
training  module  (Long,  Miller,  and  Garlichs,  1980b).  The  2-D 
simulator  was  designed  to  include  the  following  components. 

•  DEC  LSI -11  microcomputer 

•  CRT  video  terminal 

•  Floppy  disc 

•  Hard'  disc 

•  Pr  i nter  (Tl  743  RO) 

•  Trainee  video  terminal 
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•  Trainee  response  panel 

•  Random  access  T5m:r  slide  projector 

Sev  i  I  I  e-3urtek  developed  a  5-D  simulator  for  both  the 
automotive  tasks  and  the  missile  tasks.  The  automotive  training 
module  was  a  stylized,  full  scale  mock-up  of  the  NHC-250  Diesel 
Engine.  The  missile  system  module  was  a  stylized,  full  scale  mock-up 
of  the  radar  transmitter  assembly  with  a  simulated  housing  and  ail 
panel  assemblies.  (See  Table  3  for  a  summary  of  features). 

ii.  Evaluation  Plan.  Sev i I  I e-Burtek  did  not  present  a 
systematic  evaluation  plan  in  its  reports.  However,  there  were 
references  in  the  reports  to  the  evaluation  of  their  device. 

The  Sev i I  I e-Burtek  recommendation  was  to  develop  a  block  of 
instruction  for  the  simulator.  Two  groups  would  be  trained  using  this 
block  of  instruction,  an  experimental  group,  and  a  control  group.  The 
experimental  group  would  use  the  simulator  in  training  while  the 
control  group  would  train  using  the  equivalent  actual  equipment 
trainer. 


iii.  Economic  Assessment.  Sev i I  I e-Burtek  developed  a 
Logistic  Support  cost  (LSC)  model  for  estimating  life  cycle  costs 
which  was  based  on  the  AFLC  Logistics  Support  cost  model.  The  model 
is  comprised  of  five  equations  dealing  with  equipment  replacement, 
equipment  maintenance,  inventory  management,  and  cost  of  support 
equipment.  Total  life  cycle  costs  are  obtained  by  adding  the 
Logistics  Support  cost  to  the  original  acquisition  cost. 

Sev i I  I e-Burtek  recommended  exercising  the  model  a  minimum  of 
every  three  months  in  order  to  take  advantage  of  more  specific 
information  concerning  the  composition  of  the  devices.  Sev i I  I e-Burtek 
did  not  actually  solve  the  model  they  developed  since  they  did’  not 
feel  there  was  sufficient  information  to  input  to  the  model. 


Summary  of  Contractors'  FEA  Procedures 


This  section  summarizes  the  foregoing  discussion  of  contractor 
FEA  procedures.  All  four  contractors  used  similar  methods  for 
conducting  the  task  analysts,  fidelity  analysis,  and  the  training 
requirements  analysis.  In  general,  the  contractors  reviewed 
information  available  at  schools  such  as  applicable  manuals,  the 
Programs  of  Instruction,  and  interviewed  subject  matter  experts.  With 
regard  to  the  task  analysis,  the  contractors  enumerated  the  popu'ation 
of  tasks  and  classified  them  into  common  categories.  In  general,  the 
contractors  identified  similar  categories  such  as  Inspect, 
Troubleshoot,  Remove,  Replace,  Install,  Adjust,  and  Test.  The 
procedures  for  assigning  tasks  to  categories  relied  most  heavily  on 
expert  judgment. 

In  selecting  tasks  from  the  categories  for  testing  on  the 
prototype  devices,  the  contractors  were  guided  by  considerations  of 
both  cost  and  training  effectiveness  in  a  simulation  training  mode. 
Once  again,  the  contractors  relied  heavily  on  expert  judgment  which 
was  guided  by  the  contractors  philosophy  or  beliefs  concerning  how  the 
device  would  be  used.  There  appeared  to  be  differences  among  the 
contractors  in  their  philosophies. 


Grumman,  for  instance,  explicitly  saw  the  devices  as  being 
supplementary  to  existing  training  and,  therefore,  they  selected  high 
training  priority  tasks  not  currently  being  trained.  Honeywell 
implicitly  viewed  the  devices  as  being  both  supplementary  to  existing 
training  in  some  instances  and  replacing  it  in  others.  Therefore, 
they  selected  some  tasks  currently  being  trained  and  others  not  being 
trained.  Seville  saw  the  devices  as  replacing  current  training,  and 
therefore,  they  selected  tasks  currently  being  trained.  Hughes 
apparently  was  under  the  impression  that  the  training  tasks  would  be 
identified  in  AMTESS  II,  so  they  did  not  select  a  complete  set  of 
tasks  for  training  on  their  devices. 


Por  the  training  requirements  analysis,  the  contractors  relied 
on  their  review  of  appropriate  manuals  and  interviews  with  subject 
matter  experts  to  identify  particular  training  needs,  and  procedures. 
Rased  on  these,  they  developed  appropriate  programs  of  instruction  to 
train  the  tasks  they  had  selected  by  means  of  the  training  device. 

With  regard  to  the  fidelity  analysis,  the  contractors  followed 
the  same  procedures.  First,  they  identified  key  factors  of  fidelity 
necessary  for  effective  training  maintenance  tasks.  Next,  they 
examined  the  maintenance  tasks  with  regard  to  fidelity  factors  to 
determine  the  degree  of  fidelity  required.  In  contrast  to  the  task 
analysis  where  the  contractors  used  similar  categories,  the 
contractors  differed  widely  on  the  major  fidelity  factors.  Despite 
the  differences  in  fidelity  factors  examined,  despite  the  fact  that 
the  contractors  made  decisions  independently  based  primarily  on  their 
own  expert  judgment,  and  despite  the  fact  that  the  contractors  were 
evaluating,  for  the  most  part,  different  tasks,  their  conclusions  were 
remarkably  similar.  They  all  agreed  that  most  tasks  could  be  trained 
effectively  using  primarily  2-dImensional  media. 

F.  DISCUSSION 

Thi-s  section  provides  a  critical  examination  of  the  document 
review  activity  undertaken  in  this  effort.  This  examination  is  made 
within  the  context  of  the  following  four  criteria  which  were  described 
in  Section  D: 

•  Clarity  of  Requirements 

•  Completeness  of  Performance 

•  Effectiveness  of  Coordination 

•  Preparation  for  AMTESS  ll/SIMTRAIN 

1.  Clarity  of  Requirements 

The  AMTESS  I  RFP  provided  the  stimulus  for  the  entire  AMTESS 
effort.  For  purposes  of  guiding  contractor  development  of  the 


traininq  devices,  the  RPP  was  clear.  This  is  evidenced  by  the  fact 
that  all  four  contractors  developed  similar  systems  (in  that  they  were 
computer-driven,  modular,  flexible,  and  provided  for  hands-on 
training).  The  primary  differences  in  the  systems  were  the  soecific 
features  of  the  devices  such  as  type  of  storage  medium,  type  of  video 
display,  and  specific  equipment  brand. 

Nonetheless  there  were  two  areas  where  the  RFP  lacked  clear 
guidance.  One  area  was  criteria  for  task  selection.  The  RFP 
prescribed  a  standard  task  analysis,  yet  allowed  that  "Justification 
for  task  classifications  and  tor  sample  selections  shall  be  provided 
by  the  contractor  (p.  26)."  Given  that  over  500  tasks  were  analyzed, 
it  is  not  surprising  that  there  was  very  little  overlap  in  the  tasks 
selected  by  contractors.  (See  Table  2).  This  had  a  decidedly  negative 
effect  on  the  evaluation  of  the  AMTESS  devices. 

A  second  area  of  the  RFP  lacking  in  clarity  was  the  specific 
purposes  for  which  the  devices  were  to  be  employed.  While  t!ie  RFP  was 
clear  that  AMTESS  training  would  be  "adjunct"  to,  and  "supplementing" 
maintenance  training,  it  was  unclear  as  to  what  this  supplemental  role 
actually  meant.  That  is,  does  supplemental  entail  remediation, 
practice  on  tasks  currently  trained,  training  of  tasks  not  currently 
trained,  or  something  else?  This  lack  of  specificity  apparently 
influenced  contractor  task  selection  and  resulting  MPOl  development. 

Sev  i  I  I e-Burtek ,  for  example,  indicated  concern  about  the 
ability  to  evaluate  a  device  if  the  device  did  not  offer  an  MPOl 
equivalent  to  conventional  training.  If  an  MPOl  does  not  parallel  a 
block  of  the  current  PO I  ,  then  Sevi  I  I  e-Burtek  reasoned  that  there 
would  be  no  convenient  conventionally  trained  group  against  which  to 
evaluate  the  device.  As  a  result,  Sevi I le-Burtek  developed  an  MPOl 
that  could  be  substituted  for  a  block  of  current  instruction.  Grumman, 
on  the  other  hand,  developed  an  MPOl  which  was  supplemental  to  the 
current  training.  Honeywell's  MPOl  included  both  supplemental  and 
directly  substitutable  training,  and  Hughes  postponed  both  task 
selection  and  MPOl  development  until  Phase  II  of  the  AMTESS  effort. 
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'^ne  requirement  of  the  Rf-P  which  may  be  construed  as  vague, 
stated  that  "the  Contractor  shall  also  provide  recommendations  on  how 
the  breadboard  model  in  his  proposed  concept  can  be  evaluated  during 
the  demonstration  and  validation  phase  (p.  2P)."  This  statement 
elicited  varying  contractor  responses  ranging  from  Sev i I  I e-Burtek ' s 
brief  discussion  of  evaluation  to  Hughes'  considerably  more  elaborate 
presentation.  Since  the  government  had  agreed  to  have  an  independent 
agency  (API)  evaluate  the  devices,  there  appears  to  have  been  no  need 
+o  saddle  the  contractors  with  this  extra  requirement. 

2.  Completeness  of  Performance 

This  section  provides  a  discussion  of  the  activities  of  the 
contractors.  It  Includes  consideration  of  the  feasibility  study 
conducted  by  Seville,  as  well  as  the  AMTESS  I  efforts  performed  by 
Grumman,  Honeywell,  Hughes,  and  Sev 1  I  I e-Burtek . 

a.  Seville  Feasibility  Study.  The  Seville  feasibility  study 
was  one  of  the  first  steps  in  the  development  of  the  AMTESS  devices. 
It  documented  the  feasibility  of  reduced  physical  fidelity  for 
maintenance  training  as  well  as  the  feasibility  to  develop  potential 
reduced  physical  fidelity  designs.  It  also  pointed  the  way  toward 
maintenance  tasks  most  susceptible  to  reduced  physical  fidelity 
tra 1 n i ng . 

the  evaluation  m-elliodology  described  by  Seville  was  a 
straightforward  experimental  design.  In  hindsight,  one  could 
criticize  the  plan  for  making  no  reference  to  potential  problems  which 
might  result  from  conducting  a  field  experiment.  The  plan  made  no 
reference  to  the  time  commitment  and  troop  suoport  requirements  which 
would  be  required  from  field  agencies. 

Despite  this  shortcoming,  the  Seville  evaluation  plan,  was,  for 
the  most  part,  responsive  to  the  objectives  of  the  procurement  and  an 
important  first  step  in  the  development  of  the  AMTESS  devices. 
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b.  AMTESS  I  Contractor  Efforts.  the  AMTESS  I  RFP  outlined  a 


sizable  effort.  Accordinq  to  the  ISO  model,  front-end  analysis 
activities  are  essential  for  the  development  of  an  effective  training 
device.  This  section  provides  a  comment  on  the  contractors' 
performance  in  response  to  the  RFP.  This  evaluation  does  not  include 
the  qua  I i ty  of  the  contractors'  performance.  This  purpose  would 
require  data  collection  and  analysis  beyond  the  scope  of  the  present 
effort.  What  this  section  provides  is  an  indication  of  the 
comp  I eteness  of  the  contractors'  performance.  Table  4  contains  an 
indication  of  the  completeness  of  each  contractors'  performance  on  the 
major  tasks  (or  subtasks)  requested  in  the  RFP. 

As  the  table  shews,  Grumman,  Honeywell,  and  Sev i I  I e-Burtek  were 
generally  very  responsive  to  all  RFP  requirements.  The  single  area 
not  addressed  by  Sev i I  I e-Burtek  was  the  development  of  an  evaluation 
plan  for  their  device.  However,  Sev i I  I e-Bu^tek  was  cognizant  that  an 
evaluation  would  be  conducted,  and  they  attempted  to  develop  a  device 
which  lent  itself  to  easy  evaluation.  Moreover,  Seville  had  prepared 
an  evaluation  plan  in  its  feasibility  study  which  ostensibly  would 
apply  to  the  device  they  planned  to  design  in  Phase  II. 

Only  Hughes,  of  the  four  contractors,  failed  to  respond 
completely  to  a  number  of  RFP  requirements.  Hughes,  for  instance, 
devoted  only  five  pages  of  text  to  the  task  analysis.  Honeywell,  in 
contrast,  devoted  48  pages  of  text  and  approximately  70  pages  of 
appendices  to  the  task  analysis.  As  a  result  of  an  incomplete  task 
analysis,  Hughes  did  not  develop  an  MPOl  nor  did  It  develop  a  concrete 
■^-D  training  module.  These  activities  presumably  were  to  be  postponed 
until  the  AMTESS  I  I -effort  was  awarded. 

As  for  the  other  three  contractors  (Grumman,  Honeywell,  and 
Sev i I  I e-Burtek ) ,  the  completeness  of  their  design  appeared  to  allow 
for  the  construction  of  comprehensive  and  detailed  training  devices  in 
the  AMTESS  II  effort. 
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It  is  interest inp  to  note  that,  while  Huphes  was  the  least 
responsive  contractor  in  many  respects,  it  was  the  most  responsive  in 
develooing  an  evaluation  plan.  Moreover,  Huphes  was  the  first 
organization  to  point  out  the  major  potential  'opistical  problems  of 
availability  of  subjects  in  the  evaluation  phase. 

t.  Effectiveness  of  Coordination 

A  previous  report  (Criswell,  Unger,  Swezey  and  Hays,  1983) 
contained  summarized  comments  by  representatives  of  agencies  involved 
in  the  AMTESS  effort  concerning  interagency  coordination.  These 
comments  indicated  mixed  opinions  concerning  effectiveness  of  that 
coordination.  However,  one  comment  about  the  Joint  Study  Advisory 
Group  and  Joint  Working  Group  is  particularly  pertinent.  The  Task  I 
report  states  that  "one  interviewer  commented  that  the  JWG  was 
stronger  early  in  AMTESS  than  in  the  later  time  period"  (p.24). 

PM  TRADE,  the  agency  with  the  locus  of  responsibility  for  the 
effort,  and  USATSC  which  was  involved  from  the  early  stages  of  the 
effort,  both  took  a  number  of  appropriate  steps  to  insure  effective 
coordination  of  the  project.  When  the  feasibility  of  the  reduced 
physical  fidelity  trainers  was  established  by  Seville,  PM  TRADE  and 
USATSC  began  to  actively  involve  other  agencies  in  the  AMTESS  effort. 
Various  Army  schools  were  invited  to  present  training  requirements  and 
to  become  members  of  the  JSAG.  ARI  was  identified  as  the  agency 
responsible  for  evaluating  the  prototype  devices,  and  was  invited  to 
joi n  the  JSAG. 

All  JSAG  members  were  invited  to  provide  input  to  the  AMTESS  I 
RFP  and  to  evaluate  AMTESS  I  proposals.  Instructions  were  provided  to 
contractors  regarding  how  to  interact  with  various  government  agencies 
(primarily  Army  schools),  such  as  whom  to  contact,  what  to  expect,  and 
what  not  to  expect.  Periodic  JSAG/ JWG  meetings  were  held  to  keep 
participants  informed  and  involved.  Moreover,  Memorandums  ot 
Understanding  (MOU)  were  circulated  describing  the  roles  and  duties  of 
government  agencies.  These  MOUs  requested  that  agencies  formalize 


their  relationships  via  appropriate  signatures. 

While  interagency  coordination  appeared  relatively  effective 
during  the  timeframe  covered  by  this  report,  there  are  some  obvious 
reasons  why  it  may  have  become  less  effective  (as  suggested  by  some 
AMTESS  participants)  later  in  the  project.  First  -  Table  5  shows, 
■'■he  number  of  agencies  involved  in  AMTESS  multiplieo  rapidly  during 
the  first  two  years  of  the  effort.  Second,  while  the  number  of 
agencies  proliferated,  considerable  personnel  turbulence  of  AMTESS 
participants  occurred.  As  Criswell,  Unger,  Swezey  and  Hays  (19S3) 
point  out,  only  2  of  the  16  people  who  attended  the  August  1978  JSAG 
meeting  were  active  through  delivery  and  eva'ualion  of  the  prototype 
devices.  The  personnel  turbulence  issue  could  have  been  ameliorated 
had  formal  minutes  of  the  JSAG/JWG  meetings  been  recorded  and 
maintained.  However,  the  only  records  of  these  meetings  appear  in 
Memorandums  for  Record  submitted  by  individual  members  to  their  own 
organizations  but  not  necessarily  disseminated  to  or  approved  by  other 
members . 

Third,  the  timeframe  for  AMTESS  expanded  beyond  Initial 
expectations.  Indeed,  final  evaluation  of  the  devices  was  not 
completed  until  approximately  30  months  later  than  projected  in  a  1978 
schedule.  Fourth,  while  formal  MOUs  were  circulated  to  participating 
agencies,  the  process  had  shortcomings,  the  most  important  of  which 
was  the  fact  that  several  agencies  did  not  sign  MOUs.  (Since  no  fully 
signed  MOU  could  be  obtained,  it  Is  unclear  which  agencies  signed  and 
which  did  not).  If  an  agency  did  not  the  sign  an  MOU,  PM  TRADE  could 
not  necessarily  expect  that  agency  to  provide  the  necessary  support 
when  requested.  Moreover,  the  responsibilities  assigned  to  the 
agencies  named  in  the  MOUs  were  vague.  For  instance,  the  Ordnance, 
Air  Defense,  and  Missile  schools  were  given  the  task  of  providing  all 
facilities,  test  support  personnel,  and  resources  for  the  test  of  the 
prototype  devices.  However,  since  a  formal  test  plan  had  not  been 
prepared  when  the  MOU  was  circulated,  it  was  unclear  what  the  scope  of 
this  requirement  actually  entailed.  It  is  possible  that  the  vagueness 
of  the  MOU  requirement  kept  some  agencies  from  signing  it. 
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.  Preoaration  for  AMTESS  ll/SIMTRAIN 


Ultimately,  front-end  analysis  procedures  can  be  judged  in 
+erms  of  how  well  they  prepared  the  AMTESE  players  (both  government 
and  contractors)  for  actual  development  and  implementation  of  the 
devices.  table  6  lists  major  objectives  of  AMTESS  ll/SIMTRAIN  and 
shows  the  status  of  the  effort  relative  to  these  objectives  at  the  end 
of  AMTESS  I  . 


Table  6 


Correspondence  Between  AMTESS  ll/SIMTRAIN  Objectives 
and  AMTESS  I  Products 


AMTESS  I  Products 

Detailed  Prototype  Designs 

General  Acceptance  Plan 

Draft,  Incomplete  Evaluation  Plan 


AMTESS  I  I /SI MTRA I M  Objectives 

Contractors  Develop  Prototypes 
Government  Accept  Prototypes 
Government  Evaluate  Prototypes 


At  the  completion  of  AMTESS  I,  at  least  two  major 
accomplishments  were  available.  The  first  was  evidence  that  the 
reduced  physical  fidelity  training  device  concept  had  applicability  to 
a  number  of  training  situations.  The  second  was  actual  detailed 
designs  concerning  how  reduced  physical  fidelity  trainers  should  be 
configured  and  how  they  would  operate. 


However,  the  government  had  not  prepared  a  detailed  plan 
tailored  to  the  AMTESS  devices  for  inspecting  and  accepting  the 
devices,  at  least  as  far  as  can  be  ascertained  from  the  AMTESS  II  RFP 
( N6n'^9-80-R-0091  )  .  The  RFP  set  forth  a  general  set  of  procedures  for 
the  initial  inspection  and  acceptance  of  the  devices  (p.  6-7;  Exhibit 
A).  These  procedures  included  timing,  general  acceptance  procedures, 
areas  within  which  testing  might  be  conducted,  and  reporting  formats 
for  documenting  the  audits.  However,  specifics  of  the  device 
acceptance  such  as  in-plant  tests  to  be  conducted  prior  to  acceptance 
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were  left  for  developmen+  after  award  of  the  AMJESS  II  contracts.  As 
Criswell,  Unaer,  Swezey  and  Hays  (1983)  strongly  recommend,  in  a 
project  such  as  this,  precise  device  specification  should  be  made 
available  to  device  contractors  and  criteria  for  device  acceptance 
should  be  "comprehensive,  precise,  and  fully  understandable  by  all 
parties."  This  level  of  specificity  was  never  achieved  resulting  in 
oroblems  both  with  the  tasks  trained  on  the  devices  and  with  device 
dependab i I ity . 

Perhaps  the  major  shortcoming  at  the  end  of  AMTESS  I  was 
absence  of  a  detailed,  written  evaluation  plan  which  was  acceptable  to 
all  government  agencies.  One  reason  for  the  lack  of  a  detailed 
evaluation  plan  can  be  traced  to  the  uncertainty  of  what  was  to  be 
evaluated.  For  instance,  when  API  was  first  requested. to  prepare  an 
evaluation  plan,  the  tasks  to  be  trained  had  not  yet  been  determined. 
As  a  result,  API  could  not  provide  a  complete  evaluation  plan.  Another 
reason  for  the  lack  of  a  detailed  evaluation  plan  was  that  most 
parties  associated  with  AMTESS  appeared  to  feel  that  the  evaluation 
would  be  a  relatively  straightforward  process.  Seville's  evaluation 
plan  presented  in  their  feasibility  study  proposed  a  classical 
experimental  design  and  raised  no  issues  concerning  implementation 
difficulties.  Neither  did  API's  initial  plan  treat  the  odds  of 
complications  in  fielding  the  evaluation  plan.  Honeywell  and  Grumman 
both  saw  evaluation  as  a  straightforward  excercise.  Only  Hughes, 
based  on  the  documents  reviewed,  foresaw  potential  problems  with  the 
evaluation. 

From  the  perspective  of  hindsight,  it  is  easy  to  identify  major 
problem  in  conducting  the  evaluation;  i.e.,  uncertainity  regarding 
which  tasks  would  be  trained  on  the  devices.  This  issue  was 
compounded  since  there  were  two  devices  and  two  separate  maintenance 
areas  being  trained.  In  effect,  there  were  potentially  four  separate 
evaluations,  each  requiring  different  subjects  and  different  criterion 


measures . 


■'"he  "tasks  trained"  problem  was  further  maqnified  by  the 
changing  of  the  POI  at  USAOC&S  and  the  switch  in  the  curriculuni  at 
USAAnS  from  a  self-paced  to  a  lock-step  method  of  presentation.  (See 
Criswell,  Unger,  Swezey,  and  Hays,  for  a  discussion  of  this 

i ssue )  • 


The  final  topic  to  be  considered  here  was  apparently  overlooked 
in  AMTESS.  Presumably  the  objective  of  the  task  analysis  was  to 
classify  common  tasks  and  then  to  select  representative  tasks  from 
each  analytic  classification.  The  intended  consequence  of  this  step 
was  that  if  the  simulator  could  effectively  train  the  selected  tasks, 
it  should  be  able  to  train  all  the  tasks  from  which  the  simulator 
tasks  were  selected.  While  this  is  a  reasonable  objective,  some 
safeguards  were  required  to  insure  that  the  tasks  selected  actually 
represented  the  category  of  tasks  from  which  they  were  selected.  The 
AMTESS  effort  did  not  attempt  to  do  this.  As  a  result,  there  is  no 
necessary  indication  that,  even  if  the  devices  were  proven  effective 
in  transfer  of  training  that  they  would  also  be  effective  if 
configured  to  train  other  tasks  from  the  same  MOSs. 

5.  Summary  of  Front-End  Analysis  Lessons  Learned.  The  AMTESS 
effort  to  date  appears  to  be  successful  (Dybas,  198'S)  and  it  will  be 
continued  and  perhaps  expanded  to  other  maintenance  areas.  However, 
in  spite  of  its  success,  it  is  clear  that  the  effort^  did  not  proceed 
as  smoothly  as  was  expected.  This  section  summarizes  some  of  the 
lessons  learned  from  AMTESS  I. 

From  our  perspective  there  were  two  -major  problems  which 
occured  during  AMTESS  I  which  hampered  the  subseguent  AMTESS  effort. 
First,  some  of  the  key  government  agencies  taking  part  in  AMTESS  did 
not  have  specific  assignments  which  they  formally  agreed  to  undertake. 
Second,  the  tasks  which  the  AMTESS  proptypes  were  to  train  were  not 
finalized  until  several  months  after  AMTESS  II  had  begun.  Some  other 
issues  in  the  AMTESS  I  effort  included  the  following; 


•  The  major  purposes  for  which  the  proto+ypes  were  to  be  used 
were  vague.  These  purposes  must  be  made  more  explicit  at  +he  outset 
of  a  procurement  like  AMTESS. 

•  The  various  study  advisory  groups  used  during  AMTESS  were  very 
useful.  To  help  coordinate  these  groups,  the  membership  should  be 
kept  only  to  necessary  agencies,  formal  agendas  should  be  used  and 
sen+  in  advance  of  meetings,  ar.d  written  minutes  of  the  meetings 
should  be  taken  and  disseminated  to  members. 

•  Evaluation  of  the  training  devices  was  not  given  a  high  enough 
priority. 

The  next  section  of  this  report  presents  some  guidelines 
concerning  the  conduct  of  FEAs  of  the  type  examined  here.  The 
guidelines  section  is  based  both  on  the  results  of  the  case  study  of 
AMTESS  I  presented  here  as  well  as  an  earlier  report  (Criswell,  Unger, 
Swezey,  and  Hays,  198T)  which  contains  a  more  extensive  review  of 
AMTESS  I  I . 


PART  I  I 


Front-End  Guidelines 


Introduction 


If  an  analysis  of  the  +wo  orototype  AMTESS  devices  indicates 
tha-l-  the  type  of  traininq  provided  hy  the  device  can  be  both  cost-  and 
■*Ta  i  n  i  nq-ef  fecti  ve,  then  it  is  likely  that  the  Army  will  seek  to 
improve  these  devices  and  procure  additional  (or  similar)  devices. 
Roth  government  agencies  and  con+ractors  involved  in  future  efforts  of 
this  sor+  can  profit  from  an  examination  of  the  lessons  learned  from 
AMTESS  Phase  I  activities.  The  purpose  of  this  section  of  the  report 
is  to  present  lessons  learned  in  the  form  of  a  se+  of  organized 
guidelines  for  FEA  activities.  These  guidelines  are  not  comprehensive 
in  that  they  do  not  provide  a  step-by-step  list  of  procedures  to  be 
followed  during  device  development.  They  do,  however,  provide  useful 
guidance  which  may  be  applicable  to  a  wide  range  of  procurements  other 
than  AMTESS. 

Guidelines  have  been  developed  for  each  of  the  three  FEA's 
performed  by  the  contractors: 

•  Task  Analysis 

•  Traininq  Requirements  Analysis 

•  Fidelity  Ana  lysis 

In  addition,  guidelines  have  been  developed  for  analyses  which 
were  not  conducted  during  the  AMTESS  I  effort,  but  which  may  be  useful 
during  subsequent  AMTESS  efforts.  These  analyses  include: 

•  Reliability  and  Maintainability  (RAM)  Analysis 

•  Organization  Analysis 

•  Human  Factors  Analysis 

•  Person  Analysis 

Each  guideline  is  first  discussed  in  a  general  manner;  that  is, 
the  discussion  may  apply  to  a  range  of  procurement  efforts.  Following 
this  general  discussion,  the  guideline  is  discussed  in  reference  to 
the  AMTESS  effort. 
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THE  INITIATING  AGENCY  SHOULD  ASSUME  RESPONSIBILITY  FOQ 
SELECTING  TASKS. 


1  . 

a.  Pi scussion:  The  task  analytic  activities  conducted  during 
+he  front-end  of  a  training  devjce  procurement  provide  the  major  input 
to  selecting  the  particular  tasks  to  be  trained  by  the  device.  These 
activities  may  be  conducted  by  the  device  contractor  or  the 
government.  Regardless  of  who  conducts  the  task  analysis,  the 
initiating  agency  should  make  the  selection  of  tasks  to  be  trained. 
Rel i ngu i sh i no  responsibility  for  task  selection,  either  in  specifying 
task  selection  criteria,  or  in  actually  selecting  tasks,  reduces  the 
initiating  agency  ability  to  evaluate  device  contractor  performance. 

In  selecting  tasks,  the  initiating  agency  should  strive  to 
select  critical  tasks  which  are  planned  to  be  trained  in  the  same  MOS 
for  several  years  in  the  future.  It  is  further  recommended  that  if 
several  alternative  devices  are  being  developed,  they  should  train  the 
same  tasks  to  facilitate  comparison  of  the  devices. 

b.  AMTESS  Reference;  The  two  contractors  who  were 
commissioned  to  develop  training  devices  selected  the  tasks  to  be 
trained  on  the  devices.  The  differences  in  the  tasks  they  selected 
were  considerable.  This  proved  to  be  a  significant  burden  for  the 
device  evaluation.  Moreover,  it  turned  out  that  many  of  the  tasks  to 
be  trained  were  eilminated  from  the  school  curriculum  less  than  five 
months  after  they  had  been  selected.  Problems  such  as  these  could 
have  been  avoided  or  minimized  if  the  initiating  agency  had  played  a 
greater  role  in  selecting  tasks  to  be  trained. 


C.  Traininq  Requirements  Analysis  Guidelines 

1.  WHEN  FEASIBLE,  SOLICIT  THE  OPINIO.NS  OF  SEVERAL  (RATHER 
THAN  ONE)  SUBJECT  MATTER  EXPERTS  (SMEs). 

a.  Di  scuss  i  on :  The  principle  of  obtaining  data  from  a  panel 
of  experts  is  not  unique  to  front-end  analysis  activities.  Panels  of 
experts  are  frequently  consulted  in  a  variety  of  fields  in  order  to 
identify  differences  of  opinion  between  experts,  and  to  resolve  those 
differences.  Although  the  number  of  experts  available  is  usually 
limited  by  practical  constraints,  heavy  reliance  on  a  single  expert 
may  produce  results  which  are  incomplete  or  inaccurate,  or  which  are 
biased  by  the  expert’s  opinions  and  preferences. 

b.  AMTESS  Reference;  Both  contractors  relied  heavily  on  the 
expertise  of  a  single  SME  to  develop  the  radar  transmitter  simulator. 
This  expert  was  highly  knowledgeable  about  the  subject  matter,  having 
functioned  in  the  role  of  instructor  and  course  writer.  While  this 
expert  provided  a  great  deal  of  valuable  information  to  the  device 
contractors,  some  of  the  information  may  have  been  biased. 
Consultation  with  other  SMEs  could  have  resulted  in  different  tasks  or 
different  troubleshooting  techniques  being  included  in  the 
contractors'  plans  of  instruction. 


2. 


IDENTIFY  AND  RESOLVE  DIFFERENCES  BETWEEN  tpchnICAL 
MANUALS  AND  SUBJECT  MATTER  EXPERTS 


3.  Pi scuss ion;  Frequently,  serious  differences  exist  between 
official  procedures  as  described  by  technical  manuals  and  course 
svilabi,  and  the  procedures  that  actually  are  practiced  by  instructors 
and  students.  Exclusive  reliance  on  either  written  materials  or 
subject  matter  experts  to  develop  a  plan  of  instruction  is  likely  to 
lead  to  a  traininq  device  which  includes  unnecessary  and  inaccurate 
courseware,  and  which  omits  important  relevant  courseware.  A  thorough 
review  of  existing  written  materials  should  be  compared  with  knowledge 
obtained  from  subject  matter  experts  in  order  to  identify  conflicts 
and  gaps  in  information  required  to  develop  courseware. 

b.  AMTESS  Reference;  Although  the  Sev i I  I e-Burtek  device  at 
Aberdeen,  MD  included  instructions  for  operating  STE/ICE  (Simplified 
Test  Equ i pment/ I nterna I  Combustion  Engine),  no  instructions  were 
provided  for  the  prerequisite  skills  of  set-up  and  check-out  of  this 
device.  Apparently,  official  course  syllabi  indicate  that  students 
have  mastered  these  skills  during  a  prior  training  course.  However, 
none  of  the  students  and  few  of  the  instructors  using  the 
Sevi I  1%-Burtek  simulator  had  any  prior  experience  with  STE/lCE  and 
were  therefore  unprepared  to  operate  this  important  piece  of 
equipment.  Had  Sev i I  I e-BurteK  sought  to  determine  the  skill  level  of 
students  and  instructors  by  querying  those  individuals  instead  of 
relying  on  course  syllabuses,  the  problems  could  have  been  ayoided. 

Training  on  the  use  of  STE/ICE  on  the  Grumman  device  is  also 
less  than  optimal,  due  to  heavy  reliance  on  technical  manuals. 
Although  the  Grumman  device  teaches  set-up  and  check-out  of  STE/lCE, 
procedures  for  troubleshooting  with  this  equipment  are  derived  from 
inappropriate  technical  manuals.  Consultation  with  subject  matter 
experts  could  have  resulted  in  a  simple  solution  to  this  problem. 

Responsibility  for  the  STE/ICE  problems  must  also  be  accepted  by 
government  agencies.  The  requirement  to  -incorporate  this  piece  of 
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fA'-w; 


«quipmen+  into  the  MPO I  came  wel •  after  front  end  analysis  activities 
had  been  initiated.  Had  the  device  coniractors  been  forewarned  of  the 
'itE/ICE  requirement,  repercussions  could  have  been  minimized. 


nETERMINE  IN  ADVANCE  THE  SPECIFIC  TRAINING  REQUIREMENT 
TO  WHICH  A  DEVICE  WILL  BE  ADDRESSED 


■S. 

a.  Discussion:  In  developing  training  requirements,  it  is 
imperative  that  consideration  be  given  to  the  specific  requ i rement{ s) 
to  be  satisfied  by  a  •'•raining  device.  This,  to  some  extent,  is 
compatible  with  the  task  specification  requirement  stated  earlier. 
Devices  should  be  designed  to  fill  known  training  gaps  and  these  gaps 
should  be  specified  in  advance. 

b.  AMTESS  Reference:  In  the  course  of  the  AMTESS  device 
development  activities,  the  contractors  were  allowed  to  select  not 
only  the  tasks  to  be  trained,  but  also  the  training  use  to  which  their 
device  would  be  applied.  Consequently,  one  device  ( Sev i I  I e-Burtek ) 
was  designed  to  repla-.c  (or  substitute  for)  the  use  of  actual 
equipment  in  the  training  (.urriculum,  while  the  other  (Grumman)  was 
designed  to  be  an  adjunct  device  which  provides  supplemental  training, 
■^h  i  s  caused  majo''  confusion  in  subsequent  efforts  to  evaluate  the 


THE  INITIATING  AGENCY  SHOULD  ALLOW  THE  RESULTS  OF  THE 
PIDELITY  ANALYSIS  TO  INFLUENCE  THE  DESIGN  OF  THE  DEVICE. 


1  . 

a.  Discussion;  Although  there  are  many  problems  associated 
with  contracts  inat  contain  too  few  explicit  requirements,  one  must 
also  avoid  contracts  that  limit  or  reduce  the  effectiveness  of 
contractor  efforts  by  improperly  focusing  the  scope  of  work. 

b.  AMTESS  Reference:  PM  TRADE  sought  to  develop  a  simulator 
which  included  both  2-D  and  T-D  components.  The  results  of  the 
contractors'  fidelity  analyses,  however,  revealed  that  most  of  the 
tasks  in  the  MOSs  under  study  did  not  require  a  3D  component.  It  may 
have  been  more  aporopriate  for  the  contractors  to  design  a  simulator 
tha"*"  did  not  include  a  3-D  component,  or  to  analyze  other  MOSs  in  an 
attempt  to  identify  tasks  that  were  more  suitable  to  3-D  simulation. 


?.  DETERMINE  EIDELITY  REOUIREMENTS  BY  EXAMINING  THE  RANGE 
OF  VARIABLES  ENCnUMTERED  IN  TRAINING  ENVIRONMENTS. 

a.  Pi  scuss i on :  Hays  (1QR1)  ‘dentifies  numerous  variables 
which  interact  with  simulator  fidelity  to  produce  given  levels  of 
training  effectiveness.  These  variables  include;  task  type,  task 
difficulty,  specific  skills  required  to  perform  a  task,  trainee 
sophistication,  stage  of  training,  training  context,  incorporation  of 
a  device  into  a  plan  of  instruction,  user  acceptance,  and  the  use  of 
instructional  features. 

Although  resources  are  usually  too  scarce  to  examine  all  of 
these  variables  exhaustively,  preliminary  consideration  of  these 
variables  is  usually  possible  and  Is  likely  to  lead  to  more  effective 
decisions  concerning  fidelity  of  simulation. 

b.  AMTESS  Reference;  Most  of  the  fidelity  analyses  conducted 
by  the  contractors  concentrated  on  the  type  of  task  to  be  performed, 
task  difficulty,  and  the  specific  skills  required  to  perform  the  task. 
Although  these  are  critical  -^actors,  examination  of  other  variables 
described  above  also  seems  warranted. 


E.  Reliability  and  Maintainability  (RAM)  Analysis  Guidelines 

1.  CONDUCT  A  RAM  ANALYSIS  TO  PREDICT  FREQUENCY  AND  TYPE  OF 
MAINTENANCE  REQUIREMENTS  FOP  HIGH-TECHNOLOGY  EQUIPMENT. 

a.  Discussion:  Reliability  refers  to  the  probability  that  a 
system  will  be  free  of  failure  for  a  prescribed  period  of  time  under 
prescribed  com'  ions  of  performance.  A  common  quantitative  measure 
of  reliability  is  the  probability  of  failure-free  operations  for  each 
million  hours.  A  second  widely  used  measure  of  reliablity  is  mean 
time  between  failures  (MTBF)  whic'i  is  calculated  by  dividing  total 
system  operating  time  by  the  number  of  system  failures. 

Maintainability  refers  to  the  ease  with  which  maintenance 
procedures  are  performed  on  a  system.  A  common  quantitative  measure 
of  maintainability  is  ■'■he  mean  time  to  repair  (MTTR). 

A  related  concept,  availability,  refers  to  the  probability  that  a 
system  will  be  operating  at  any  particular  time.  It  can  be  defined 
as;  MTBF 

MTBF  +  MTTR 

where:  MTTR  =  mean  time  to  repair,  and 

MTBF  =  mean  time  between  failures 

Although  there  are  a  variety  of  other  measures  of  reliability 
and  maintainability  they  need  not  be  addressed  here.  The  point  to  be 
made  is  that  some  estimate  needs  to  be  made  of  the  types  of 
maintenance  activities  t’ at  will  be  reouired  to  keep  the  system 
operational.  If  these  estimates  can  be  made  by  the  contractor  during 
early  development  phases  of  the  effort,  then  assessments  can  be  made 
regarding  the  acceptability  of  these  failure  rates.  If  these  rates 
seem  unacceptable,  then  engineering  changes  can  be  made  early  in  the 

effort  to  alleviate  the  problem.  In  addition,  the  initiating  agency 
cSn  make  concrete  plans  for  servicing  the  system  if  it  knows; 
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•  +he  componen+s  th3+  are  mos+  likely  to  fail, 

•  the  type  of  maintenance  personnel  required  to  service  the 
system,  and 

•  maintenance  procedures. 

b.  AMTESS  Reference:  Use  of  AMTESS  devices  could  have  been 
enhanced  if  effective  RAM  analyses  had  been  conducted.  However,  since 
useful  estimates  of  system  reliability  were  not  available,  the  schools 
had  little  notion  of  how  well  the  devices  would  perform.  If  estimates 
of  reliability  had  been  available,  troops  may  not  have  been  expected 
to  receive  training  on  the  device  within  confined  time  restrictions. 
Also  evaluation  of  the  devices  could  have  been  planned  more 
effectively  if  it  had  been  known  how  often  the  devices  would  fail. 

A  maintainability  analysis  could  have  indicated  the  skills 
required  to  service  the  AMTESS  devices.  Had  this  information  been 
known,  users  could  have  determined  if  they  could  repair  the  device 
themselves,  or  if  contractor  representatives  would  be  necessary. 
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2. 


PROVIDE  SCHEMATICS  OF  THE  DEVICE  TO  THE  USER 


a.  Pi scussion:  The  number  of  people  who  can  effectively 
troubleshoot  e  comolex  piece  of  equipment  is  significantly  increased 
when  schematics  are  available.  When  schematics  are  not  available,  the 
ability  to  repair  equipment  may  be  limited  to  those  individuals  who 
developed  the  equipment. 

b.  AMTESS  Reference:  The  AMTESS  devices  may  have  been 
maintained  more  effectively  had  schematics  been  available  to  users. 
Since  they  were  not  available,  users  were  forced  to  rely  on  the  device 
manufacturers  for  maintenance  support. 


^OMOLICT  an  analysis  to  determine  the  ENVIRONMENTAL 

RIGORS  TO  WHICH  THE  DEVICE  WIlL  BE  EXPOSED 


a.  Pi scussion:  Althouqh  proto+yoes  cannot  be  expected  to  be 
as  durable  as  production  models,  they  must  be  durable  enough  to 
withstand  manual  use  during  an  evaluation  period. 

b.  AMTESS  Reference:  Had  the  contractors  considered  the 
environment  in  which  the  prototypes  would  be  operated,  the  prototypes 
could  have  been  designed  to  operate  much  more  reliably.  For  example, 
it  was  suspected  that  the  Grumman  device  at  Fort  Bliss  failed 
repeatedly  because  of  excessive  dust  in  the  operating  environment. 
Dust  storms  are  common  in  the  spring  In  West  Texas.  It  seems  likely 
that  a  RAM  analysis  could  have  anticipated  and  solved  this  problem  by 
either  building  the  device  to  withstand  the  dust  or  describing 
preventative  measures  that  could  have  been  taken  to  reduce  dust 
infiltration.  Similarly,  problems  with  the  Grumman  videodisc  arose 
because  instructors  handled  the  disc  with  dirty  hands.  Procedures 
could  have  been  developed  to  avoid  these  problems  if  they  had  been 
identified  in  a  RAM  analysis.  Further,  steps  should  be  taken  to 
ensure  that  device  contractors  are  cognizant  of,  and  in  compliance 
with,  all  relevant  military  specifications  which  are  applicable  to  the 
device  under  procurement. 


77 


F.  Organization  Analysis  Guidelines 

1.  conduct  an  organization  analysis  to  ensure  that  the 

GOALS  OF  PARTICIPATING  ORGANIZATIONS  ARE  COMPATIBLE  WITH 
THE  INITIATING  AGENCY'S  GOALS  FOR  THE  DEVICE. 

a.  Di scussion;  In  order  for  a  training  device  or  system  to  be 
effective,  consideration  must  be  given  not  only  to  the  immediate 
organization  in  which  it  resides,  but  also  to  the  larger 
organizational  environment.  Since  the  training  device  does  not 
operate  in  a  vacuum,  it  must  be  designed  so  that  It  fits  into  a 
particular  organization  or  designed  with  flexibility  so  that  it  can 
fit  into  a  variety  of  organizations.  Identification  of  conflicting 
goals  is  an  essential  part  of  such  an  organization  analysis. 

b.  AMTESS  Reference:  Several  examples  from  the  AMTESS  effort 
serve  to  illustrate  the  importance  of  specifying  an  organization's 
goals.  First,  It  appears  that  one  of  the  important  goals  of 
instructors  in  the  wheeled  vehicle  repair  unit  at  Aberdeen,  MD  was  to 
train  soldiers  as  quickly  as  possible.  Duality  of  instruction  was  as 
important  as  speed  of  instruction  because  the  unit  was  under  a  great 
deal  of  pressure  to  graduate  large  numDers  of  trainees  on  time.  Since 
this  was  the  case,  instructors  had  little  time  to  train  students  on 
how  to  operate  fhe  Sev i I  I e-Burtek  simulator.  Students  frequently 
received  less  than  15  minutrs  of  instruction  on  how  to  operate  the 
device,  even  though  a  one  to  two  hour  introduction  to  the  simulator  Is 
required.  Consequently,  students  received  far  less  than  optimal 
training  on -the  device  since  they  could  not  operate  it  properly.  Had 
an  organization  analysis  been  conducted,  the  importance  of  training 
time  could  have  been  discovered  and  the  courseware  could  have  been 
designed  accordingly. 

An  organization  analysis  may  also  have  revealed  changing 
attitudes  concerning  self-paced  versus  lock-step  instruction.  The 
Grumman  device  is  relatively  inflexible  In  terms  of  its  ability  to 
adapt  to  a  change  in  instructional  techniques  or  POI.  An  organization 
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2.  CONDUCT  AN  OPGANIZATION  ANALYSIS  TO  ENSURE  THAT  THE 

RESOlf(CES  OF  THE  USER  ORGANIZATION  ARE  COMPATIBLE  WITH 
DEVICE  REOUIREMENTS. 

e.  Pi scussion:  Thi^  quidellnp  is  similar  to  the  one  described 
above  in  that  it  seeks  to  ensure  that  the  training  device  and  user 
organization  are  compatible.  In  this  case,  the  point  is  to  determine 
if  the  training  system  will  require  resources  which  the  user 
organization  cannot  deliver.  Here,  resources  may  refer  to  such  things 
as  manpower,  facilities,  support  equipment,  and  procedures. 

b.  AMTESS  Reference:  Both  AMTESS  devices  were  designed  to 
allow  an  instructor  to  teach  one  or  two  students  at  a  time.  However, 
the  student- i nstructor  ratio  for  conventional  training  in  the  wheeled 
and  tracked  vehicle  repair  MOSs  at  Aberdeen,  Md .  is  typically 
six-to-one,  and  occasionally  higher.  AMhough  special  arrangements 
were  made  (with  great  difficulty)  to  evaluate  the  device  using  a  two- 
to-one  ratio,  the  devices  could  not  be  used  unit-wide  with  such  a 
ratio  since  there  is  a  severe  shortage  of  qualified  instructors.  An 
organization  analysis  could  have  identified  this  problem  and  helped  to 
design  the  device  so  as  to  minimize  this  problem. 


G.  Human  Factors  Analysis  Guidelines 


1.  CONDUCT  A  HUMAN  FACTORS  ANALYSIS  tq  ENSURE  THAT  THE 
DEVICE  IS  EASY  TD  OPERATE. 

a.  Pi scussion:  The  increasing  complexity  of  weapon  systems  is 
reflected  in  the  complexity  of  weapon  system  training  devices. 
Complexity  of  the  task  which  the  device  performs  may  or  may  not  be 
desirable  deoending  on  training  regui rements .  However,  students 
should  not  have  to  learn  a  complex  set  of  procedures  simply  to  operate 
the  training  device.  The  effectiveness  of  the  device  is  likely  to 
increase  if  ease  of  use  is  increased. 

h.  AMTESS  Reference:  Although  Sev i I  I e-Burtek  attempted  to 
simplify  the  operation  of  their  device  through  the  use  of  a  student 
responder  panel,  this  panel  was  confusing  to  students.  In  addition, 
s+udents  also  were  required  to  look  at  a  slide  projector  unit,  a  CRT, 
a  T-D  module,  technical  manuals,  and  test  equipment.  Similar  problems 
occurred  wi+h  the  Grumman  device's  video-disc  capability.  Many 
students  seemed  overwhelmed  by  all  of  the  information  that  was 
presented.  Had  the  devices  been  pretested  with  pilot  students  prior 
to  delivery,  +hey  may  have  been  redesigned  in  order  to  make  them 
easier  to  operate. 
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CONDUCT  A  HUMAN  "ACTORS  /ANALYSIS  TO  ENSURE  A 

satisfactory  man-machine  interface. 


3.  Pi  scuss  i  on :  There  are  a  variety  of  docijments  that  describe 
the  development  of  human  engineering  program  plans  and  human 
engineering  design  critera  (M I L-H-46855B,  1979;  MIL  STD  1472C,  19R1). 
Among  other  topics,  these  documents  provide  guidelines  for  designing 
controls,  displays,  crew  stations,  and  individual  work  stations.  The 
guidance  provided  in  these  documen+s  is  appropriate  to  the  design  of 
training  devices.  Failure  to  use  human  engineering  principles  in  the 
design  of  a  training  device  may  result  in  a  wide  variety  of  problems 
which  will  degrade  the  effectiveness  of  the  device. 

b.  AMTESS  Reference!  Student  performance  on  the  Grumman 
device  at  Aberdeen  could  have  been  improved  if  human  engineering 
prlncioles  had  been  incorporated  into  the  device.  For  example, 
students  were  reguired  to  identify  and  use  a  number  of  cables  attached 
to  the  STE/ICE  device.  However,  since  the  cables  were  not  labelled 
(as  they  are  on  the  operational  equipment),  students  spent  excessive 
amounts  of  time  attempting  to  Identify  the  cables  that  were 
appropriate  for  each  procedure. 

Students  also  reported  that  the  CRT  on  the  Grumman  device  and 
the  slide  projector  unit  on  the  Sev i I  I e-Burtek  device  were  too  small, 
i.e.,  details  of  the  materials  presented  could  not  be  identified. 
Further,  the  Grumman  CRT  was  too  low  to  be  viewed  comfortably  while 
standing,  and  since  neither  the  Grumman  CRT  nor  the  Sev i I  I e-Burtek 
slide  projector  unit  swiveled,  students  could  not  look  at  the  CRT 
while  they  were  working  on  the  3-D  module.  Problems  such  as  these 
could  have  been  identified  and  corrected  following  a  human  factors 
ana  lysis. 
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H.  Person  Analysis  Guideline 


1.  CONDUCT  A  PERSON  ANALYSIS  TO  IDENTIFY  PERTINENT  SKILL 
LEVELS  OF  THE  TARGET  POPULATION. 

a.  Pi scussion:  The  previous  puidelines  on  human  factors 
analysis  stressed  the  importance  of  designing  a  device  which 
incorporates  basic  principles  of  human  engineering.  In  addition,  it 
is  necessary  to  design  and  to  accommodate  certain  pertinent  readiness 
skill  levels  of  the  target  population.  An  example  of  such  a  skill 
would  be  reading.  Another  example  might  be  general  task  attention. 
Selection  of  readiness  skills  to  assess  in  a  person  analysis  might  be 
based  on  discussion  with  instructors  and  direct  observation  of  ongoing 
tra i n i ng  . 

b.  AMTESS  Reference;  The  AMTESS  contractors  completed 
portions  of  a  person  analysis.  Instructors  and  evaluators  reported 
that  the  reading  level  of  the  materials  was  appropriate  for  the  target 
popu I  at  ion . 

With  respect  to  general  task  attention,  however,  some  problems 
occurred  during  device  testing.  Many  instr^jctors  noted  that  the 
devices  did  not  hold  students'  atrention  well.  Special  attempts  at 
holding  student  attention  may  be  particularly  important  in  military 
training.  In  the  tracked  vehicle  repair  course  at  Aberdeen,  ^D, 
students  were  observed  exploiting  the  HELP  feature  of  the  device  in 
order  to  leave  the  device  as  guickly  as  possible  each  day.  Students 
discovered  that  they  could  rapidly  complete  instructional  sequences  by 
freguently  using  the  HELP  feature  which  would  quickly  tell  them 
answers  to  questions.  Thus,  students  did  not  have  to  search  for 
answers  themselves.  Speedy  progress  is  important  in  a  training 
course,  but  not  at  the  expense  of  omitting  crucial  training. 
Apparently,  the  actual  instructional  progress  of  some  students  was 
mitigated  because  the  device  failed  to  hold  their  attention. 
Observation  of  students  in  training  prior  to  device  construction  might 
have  pointed  out  special  needs  for  holding  students'  attention. 


I.  SUMMARY  OF  GUIDELINES 


The  front  end  analysis  guidelines  described  in  the  oreceeding 
sections  are  summarized  below: 

1 .  TASK  ANALYSIS  GUIDELINE 

The  initiating  agency  should  assume  responsibility  for 
selecting  tasks. 

2.  TRAINING  REQUIREMENTS  ANALYSIS  GUIDELINES 

a.  When  feasible,  solicit  the  opinions  of  several  (rather  than 
one)  subject  matter  experts  (SMEs). 

b.  Identify  and  resolve  differences  between  technical  manuals 
and  subject  matter  experts. 

c.  Determine  in  advance  the  specific  training  reguirement  to 
which  a  device  will  be  addressed. 

FIDELITY  ANALYSIS  GUIDELINES 

a.  The  initiating  agency  should  allow  the  results  of  the 
fidelity  analysis  to  influence  the  design  of  the  device. 

b.  Determine  fidelity  requirements  by  examining  the  range  of 
variables  encountered  in  training  environments. 

4.  reliability  and  MAINTAINABILITY  (RAM)  ANLAYSIS  GUIDELINES 

a.  Conduct  a  RAM  analyis  to  predict  frequency  and  type  of 
maintenance  requirements  for  high-technology  equipment. 

b.  Provide  schematics  of  the  device  to  the  user. 


c.  Conduct  a  RAM  analysis  to  determine  the  environmental 
rigors  +o  which  the  device  will  be  exposed. 

■5.  ORGANIZATION  ANALYSIS  GUIDELINES 

a.  Conduct  an  organization  analysis  to  ensure  that  the  goals 
of  Dart ici pat i ng  organizations  are  compatible  with  the  initiating 
agency's  qoals  tor  the  device. 

b.  Conduct  an  organization  analysis  to  ensure  that  the 
resources  of  the  user  organization  are  compatible  with  the  device 
requ i rements . 

6.  HUMAN  FACTORS  ANALYSIS  GUIDELINES 

a.  Conduct  a  human  factors  analysis  to  ensure  that  the  device 
is  easy  to  operate. 

b.  Conduct  a  human  factors  analysis  to  ensure  a  satisfactory 
man-machine  interface. 

7.  PERSON  ANALYSIS  GUIDELINE 

Conduct  a  person  analysis  to  identity  pertinent  skill  levels  of 
%  ^ 
the  target  population. 
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